A method and system for model-based fusion of pre-operative image data and intra-operative fluoroscopic images is disclosed. A fluoroscopic image and an ultrasound image are received. The ultrasound image is mapped to a 3D coordinate system of a fluoroscopic image acquisition device used to acquire the fluoroscopic image. Contours of an anatomical structure are detected in the ultrasound image, and a transformation is calculated between the ultrasound image and a pre-operative CT image based on the contours and a patient-specific physiological model extracted from the pre-operative CT image. A final mapping is determined between the CT image and the fluoroscopic image based on the transformation between the ultrasound image and physiological model and the mapping of the ultrasound image to the 3D coordinate system of the fluoroscopic image acquisition device. The CT image or the physiological model can then be projected into the fluoroscopic image.