Techniques and systems are provided for processing video data. For example, techniques and systems are provided for performing content-adaptive object or blob tracking. To perform the content-adaptive object tracking, a blob tracker is associated with a blob generated for a video frame. The blob includes pixels of at least a portion of a foreground object in a video frame. A size of the blob can be determined to be greater than a blob size threshold. The blob tracker can be converted to a normal tracker based on the size of the blob being greater than the size threshold. The associated blob tracker and blob are output as an identified blob tracker-blob pair when the blob tracker is converted to the normal tracker.