A method and system for model-based fusion of multi-modal volumetric images is disclosed. A first patient-specific model of an anchor anatomical structure is detected in a first medical image acquired using a first imaging modality, and a second patient-specific model of the anchor anatomical structure is detected in a second medical image acquired using a second imaging modality. A weighted mapping function is determined based on the first patient-specific model of the anchor anatomical structure and the second patient-specific model of the anchor anatomical structure using learned weights to minimize mapping error with respect to a target anatomical structure. The target anatomical structure from the first medical image to the second medical image using the weighted mapping function. In an application of this model-based fusion to transcatheter valve therapies, the trachea bifurcation is used as the anchor anatomical structure and the aortic valve is the target anatomical structure.