A trailer angle detection system for a vehicle towing a trailer includes a rearward viewing camera disposed at a rear portion of a vehicle, and an image processor operable to process image data captured by the camera. Responsive to processing of captured image data when the vehicle is towing a trailer, the trailer angle detection system determines the presence of the trailer at the rear of the vehicle and attached at the vehicle and determines feature vectors of the detected trailer over multiple frames of captured image data. Responsive to movement of the trailer relative to the vehicle, the trailer angle detection system tracks feature vectors for different positions of the detected trailer. The trailer angle detection system determines an angle of the trailer responsive to detection of tracking of feature vectors the trailer in the field of view of the camera over multiple frames of captured image data.