Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for segmenting a medical image. In one aspect, a method comprises: receiving a medical image that is captured using a medical imaging modality and that depicts a region of tissue in a body; and processing the medical image using a segmentation neural network to generate a segmentation output. The segmentation neural network can include a sequence of multiple encoder blocks and a decoder subnetwork. Training the segmentation neural network can include determining a set of error values for a segmentation channel; identifying the highest error values from the set of error values for the segmentation channel; and determining a segmentation loss based on the highest error values identified for the segmentation channel.