Provided are embodiments for a computer-implemented method. The method includes receiving a sequence of image data, transforming objects in each frame of the sequence of the image data into direction vectors, and clustering the direction vectors based at least in part on features of the objects. The method also includes mapping the direction vectors for the objects in each frame into a position-orientation data structure, and performing tracking using the mapped direction vectors in the position-orientation data structure. Also provided are embodiments of a computer program product and a system for performing object tracking.