Systems and methods are disclosed for performing operations comprising: receiving first and second images depicting an anatomy of a subject; obtaining a segmentation associated with the first image; applying a trained neural network to estimate the adapted segmentation corresponding to the anatomy depicted in the second image, the trained network consisting of two sub-networks: a registration sub-network, generating an initial segmentation estimate representing a deformation of the segmentation associated with the first image to fit the anatomy depicted in the second image, and a segmentation refinement sub-network, predicting a refined segmentation for the second image given the initial segmentation estimate.