A vehicular control system includes at least one processor that processes image data captured by a forward viewing camera and radar data captured by a forward sensing radar sensor. With the system controlling driving of the vehicle, the system determines a triggering event that requires driving of the vehicle to be handed over to a driver of the vehicle before the vehicle encounters an event point, and the system (i) determines a total action time until the vehicle encounters the event, (ii) estimates a driver take over time for the driver to take over control and (iii) estimates a handling time for the driver to control the vehicle to avoid encountering the event point. Based on the determined and estimated times, the system (i) requests the driver take over control of the vehicle or (ii) controls the vehicle to slow down and stop the vehicle before the event point.