A method for automatically localizing left ventricle in medical image data includes acquiring a sequence of three-dimensional medical images spanning a cardiac cycle. Each of the images includes a plurality of two-dimensional image slices, one of which is defined as a template slice. The template slice of each medical image of the sequence is automatically cropped to include the heart and a margin around the heart based on temporal variations between pixels of the template slice throughout the sequence of medical images. The template slice of each medical image of the sequence is automatically contoured to determine the endo-cardial and epi-cardial boundaries for at least the end-diastolic and end-systolic phases. Localization information is generated for the left ventricle based on the determined endo-cardial and epi-cardial boundaries for at least the end-diastolic and end-systolic phases.