A method and associated apparatus for using a trajectory-based technique to detect a moving object in a video sequence at incorporates human interaction through a user interface. The method comprises steps of identifying and evaluating sets of connected components in a video frame, filtering the list of connected components by comparing features of the connected components to predetermined criteria, identifying candidate trajectories across multiple frames, evaluating the candidate trajectories to determine a selected trajectory, eliminating incorrect trajectories through use of the interface and processing images in said video sequence responsive to the evaluating and eliminating steps.