A vision system of a vehicle includes a camera disposed at a vehicle and having a field of view exterior of the vehicle. The camera may have a wide angle lens with more distortion at side regions of the field of view than at a center region of the field of view. An image processor processes captured image data to determine gradient information of captured image data, and responsive to processing gradient information, determines a potential cross traffic object at one of the side regions of the field of view. The image processor processes multiple frames of captured image data to determine movement of an upper portion and a lower portion of a gradient of the determined potential cross traffic object and, responsive to such processing, determines if the detected potential cross traffic object is a cross traffic object that is moving into a path of the vehicle.