A method and system for adaptable video-based object tracking includes acquiring video data from a scene of interest and identifying an initial instance of an object of interest in the acquired video data. A representation of a target object is then established. One or more motion parameters associated with said scene of interest are used to adjust the size of a search neighborhood associated with said target object. The target object is then tracked frame-by-frame in the video data.