A vision system for a vehicle includes a rearward facing camera disposed at a rearward portion of a vehicle. With a trailer attached to the vehicle and rearward of the vehicle, a processor is operable to process captured images and, responsive at least in part to the processing, is operable to determine a plurality of trailer and vehicle-trailer interface parameters. The system is operable to calculate a calculated trailer angle based at least in part on calculations involving the parameters and a steering angle of the vehicle. Responsive to processing of images captured by the camera, the system is operable to determine a determined trailer angle of the trailer relative to a longitudinal axis of the vehicle. Responsive to the determined trailer angle and the calculated trailer angle, the system is operable to determine a degree of offset between the calculated trailer angle and the detected trailer angle.