A method tracks a moving object in a video acquired of a scene with a camera. A background model is maintained for each frame, and moving objects are detected according to changes in the background model. An object model is maintained for the moving object, and kernels are generated for the moving object. A mean-shift process is applied to each kernel in each frame to determine a likelihood of an estimated location of the moving object in each frame, according to the background models, the object model, and the mean shift kernels to track the moving object in the video.