A collision avoidance system of a vehicle includes a camera disposed at a vehicle and having a field of view exterior of and rearward of the vehicle. An image processor is operable to process image data captured by the camera. Responsive to image processing of captured image data, the collision avoidance system is operable to determine an approach of a vehicle rearward of the vehicle and to determine an excitation level depending on a difference between the speed of the approaching vehicle and the speed of the equipped vehicle and a distance between the approaching vehicle and the equipped vehicle. Responsive to the determined excitation level, the collision avoidance system may provide an alert to the driver of one or both vehicles, prepare for impact, control a vehicle braking system, vehicle steering system and/or vehicle acceleration system and/or communicate information to the approaching vehicle via a telematics system.