An image volume formed by plural anatomical images each having plural image slices of different imaging modalities is segmented by a 2D convolutional neural network (CNN). An individual anatomical image is preprocessed to form a mixed-context image by incorporating selected image slices from two adjacent anatomical images without any estimated image slice. The 2D CNN utilizes side information on multi-modal context and 3D spatial context to enhance segmentation accuracy while avoiding segmentation performance degradation due to artifacts in the estimated image slice. The 2D CNN is realized by a BASKET-NET model having plural levels from a highest level to a lowest level. The number of channels in most multi-channel feature maps of a level decreases monotonically from the highest level to the lowest level, allowing the highest level to be rich in low-level feature details for assisting finer segmentation of the individual anatomical image.