A tumor is tracked in multiple sequences of images acquired concurrently from different viewpoints. Features are extracted in each set of current images using a window. A regression function, subject to motion constraints, is applied to the features to obtain 3D motion parameters, which are applied to the tumor as observed in the images to obtain a 3D location of the object. Then, the shape of the 3D object at the 3D location is projected onto each image to update the location of the window for the next set of images to be processed.