A vision system of a vehicle includes at least one camera disposed at a vehicle and having a field of view exterior of the vehicle, and an image processor operable to process image data captured by the camera. Responsive to image processing of captured image data, the image processor determines objects present in the field of view of the camera. The vision system processes additional frames of captured image data to enhance determination of objects of interest. The vision system initially detects an object present in the field of view of the camera and conducts hypotheses filtering and hypotheses merging and, responsive to the hypotheses merging, the system determines that the detected object is an object of interest or determines that the detected object is not an object of interest.