A vehicular control system includes a control having a processor that processes image data captured by a forward viewing camera. When the control system is controlling driving of the vehicle, the control determines a triggering event that requires driving of the vehicle to be handed over to a driver of the vehicle before the vehicle encounters an event point. Responsive to determination of the triggering event, the control (i) determines a total action time until the vehicle encounters the event, (ii) estimates a driver take over time for the driver to take over control and (iii) estimates a handling time for the vehicle to be controlled to avoid encountering the event point, and, responsive to the determination and estimations, the control system (i) allows the driver to take over control of the vehicle or (ii) controls the vehicle to slow down and stop the vehicle before the event point.