A method and system for extracting coronary vessels fluoroscopic image sequences using coronary digital subtraction angiography (DSA) are disclosed. A set of mask images of a coronary region is received, and a sequence of contrast images for the coronary region is received. For each contrast image, vessel regions are detected in the contrast image using learning-based vessel segment detection and a background region of the contrast image is determined based on the detected vessel regions. Background motion is estimated between one of the mask images and the background region of the contrast image by estimating a motion field between the mask image and the background image and performing covariance-based filtering over the estimated motion field. The mask image is then warped based on the estimated background motion to generate an estimated background layer. The estimated background layer is subtracted from the contrast image to extract a coronary vessel layer for the contrast image.