A trailer angle detection system for a vehicle towing a trailer includes a camera disposed at a rear portion of the vehicle and having a field of view rearward of the vehicle. The camera includes a two dimensional imager array having multiple rows and columns of photosensing elements. An image processor processes image data captured by the camera. With the vehicle towing a trailer, and via processing by the image processor of image data captured by the camera, the trailer angle detection system detects a portion of the trailer that is being towed by the vehicle. The trailer angle detection system determines, via processing by the image processor of image data captured by the camera, an angle of the trailer relative to the vehicle responsive to determination of which columns of photosensing elements sense the detected portion of the trailer in the field of view of the camera.