A vehicular camera calibration system includes a camera disposed at a vehicle and having a field of view exterior of the vehicle and being operable to capture image data. Responsive to processing of image data captured by the camera, the camera calibration system determines features at the ground adjacent to the vehicle and tracks the determined features over two or more frames of captured image data. Responsive to processing of frames of captured image data during movement of the vehicle along a straight path on a planar surface, the camera calibration system is operable to determine a misalignment of the camera. The camera calibration system calibrates the misaligned camera by using an assumption that the determined features lie in a plane parallel to the planar surface along which the vehicle is traveling. The camera calibration system calibrates the camera by reducing ambiguity in vectors normal to the planar surface.