An apparatus and method for implementing object motion segmentation and object trajectory segmentation for an image sequence. Specifically, block-based motion vectors for a pair of adjacent frames are used to derive optical flow, e.g., affine, motion parameters. Such optical flow motion parameters are employed to determine key objects where their motion and trajectory within a sequence of frames are calculated and stored. Such object motion information is used to improve or offer image processing functions such as context-based indexing of the input image sequence by using motion-based information.