A method and system for tracking a needle in a fluoroscopic image sequence is disclosed. In order to track a needle in a fluoroscopic image sequence, the needle is initialized in a first frame of the fluoroscopic image sequence. Needle segments are detected in each subsequent frame of the fluoroscopic image sequence, and the needle is detected in each frame of the fluoroscopic image by tracking the needle from a previous frame of the fluoroscopic image sequence based on the detected needle segments in the current frame.