Systems and methods for estimating the centers of moving objects in a video sequence are disclose. One embodiment is a method of defining one or more motion centers in a video sequence, the method comprising receiving a video sequence comprising a plurality of frames, receiving a motion history image for each of a subset of the plurality of frames based on the video sequence, identifying, through use of the motion history image, one or more data segments having a first orientation, wherein each data segment having the first orientation has a start location and a length, identifying, one or more data segments having a second orientation, wherein each element of a data segment having the second orientation is associated with a data segment having the first orientation, and defining a corresponding motion center for one or more of the indentified data segments having the second orientation.