A system and method is provided which obtains different medical images (210) showing an anatomical structure of a patient and having been acquired by different medical imaging modalities and/or different medical imaging protocols. The system is configured for fitting a first deformable model to the anatomical structure in the first medical image (220A), fitting a second deformable model to the anatomical structure in the second medical image (220B), mutually aligning the first fitted model and the second fitted model (230), and subsequently fusing the first fitted model and the second fitted model to obtain a fused model (240) by augmenting the first fitted model with a part of the second fitted model which is missing in the first fitted model; or adjusting or replacing a part of the first fitted model based on a corresponding part of the second fitted model having obtained a better fit. The fused model represents a multimodal/multi-protocol segmentation of the anatomical structure, and provides a user with a more comprehensive understanding of the anatomical structure than known models.