A semi-automatic method of tracking color objects in a video image sequence starts by separating the objects on the basis of color and identifying an object of interest to track. A Kalman predictive algotithm in used to predict the position of the centroid of the object of interest through successive frames. From the predicted position the actual centroid is measured and the position and velocity are smoothed using a Kalman filter. Error recovery is provided in the event the centroid falls outside the field of view or falls into an area of a different color, or in the event the tracking algorithm breaks down.