A method and system for fusion of multi-modal volumetric images is disclosed. A first image acquired using a first imaging modality is received. A second image acquired using a second imaging modality is received. A model and of a target anatomical structure and a transformation are jointly estimated from the first and second images. The model represents a model of the target anatomical structure in the first image and the transformation projects a model of the target anatomical structure in the second image to the model in the first image. The first and second images can be fused based on estimated transformation.