A collision avoidance system of a vehicle includes a sensor configured to be disposed at a vehicle for sensing exterior and forwardly of the vehicle. A processor is operable to process sensor data captured by the sensor to determine the presence of a pedestrian ahead of the vehicle and at or moving towards a path of travel of the vehicle. The processor determines a time to collision based on a determined distance to the pedestrian and determined speed of the pedestrian and speed of the vehicle. The collision avoidance system is operable to generate an alert to the driver of the vehicle at a threshold time before the determined collision with the pedestrian. Responsive to a parameter, the collision avoidance system adjusts the threshold time to generate the alert at an earlier time.