A system and method for defining and tracking a deformable shape of a candidate anatomical structure wall in a three dimensional (3D) image is disclosed. The shape of the candidate anatomical structure is represented by a plurality of labeled 3D landmark points. At least one 3D landmark point of the deformable shape in an image frame is defined. A 3D cuboid is defined around the detected 3D landmark point. For each landmark point associated with the anatomical structure, its location and location uncertainty matrix is estimated in subsequent frames relative to the reference anatomical structures. A shape model is generated to represent dynamics of the deformable shape in subsequent image frames. The shape model includes statistical information from a training data set of 3D images of representative anatomical structures. The shape model is aligned to the deformable shape of the candidate anatomical structure. The shape model is fused with the deformable shape. A current shape of the candidate anatomical structure is estimated.