A method is described that includes identifying a set of features of an object, the features being tracked in an image captured by a camera. The method also includes creating a field of vectors for the reference points. The vectors indicate magnitude and direction of change in position of the reference points across more than one frame of the image. The method further includes identifying existence of out of plane movement of the object's features from same radial orientation of the vectors. The method further includes determining an amount of closer/farther movement of the object's features to/from the camera from change in distances between a plurality of the reference points. The method further includes adjusting a position of camera's lens in view of the amount of closer/farther movement of the object's features to keep the camera focused on the object.