A method and system for vessel segmentation in fluoroscopic images is disclosed. Hierarchical learning-based detection is used to perform the vessel segmentation. A boundary classifier is trained and used to detect boundary pixels of a vessel in a fluoroscopic image. A cross-segment classifier is trained and used to detect cross-segments connecting the boundary pixels. A quadrilateral classifier is trained and used to detect quadrilaterals connecting the cross segments. Dynamic programming is then used to combine the quadrilaterals to generate a tubular structure representing the vessel.