A surveillance system and method are provided. The surveillance system includes an image capture device configured to capture a video sequence of a target area that includes a set of objects and is formed from a set of image frames. The surveillance system also includes a processor. The processor is configured to detect the objects to form object detections, and track the object detections over the frames to form tracked detections. The processor is further configured to generate for a current input frame, responsive to conditions, a set of sparse object proposals for a current location of an object based on: (i) the tracked detections of the object from an immediately previous frame; and (ii) detection proposals for the object derived from the current frame. The processor is additionally configured to provide a user perceptible indication of the current location of the object, based on the sparse object proposals.