A vision system of a vehicle includes a camera and a non-imaging sensor. With the camera and the non-imaging sensor disposed at the vehicle, the field of view of the camera at least partially overlaps the field of sensing of the non-imaging sensor at an overlapping region. A processor is operable to process image data captured by the camera and sensor data captured by the non-imaging sensor to determine a driving situation of the vehicle. Responsive to determination of the driving situation, Kalman Filter parameters associated with the determined driving situation are determined and, using the determined Kalman Filter parameters, a Kalman Filter fusion may be determined. The determined Kalman Filter fusion may be applied to captured image data and captured sensor data to determine an object present in the overlapping region.