A vehicular control system includes at least one camera disposed in a vehicle and viewing a driver of the vehicle who is sitting in a driver seat of the vehicle. The vehicular control system is operable to drive the vehicle along a road in accordance with SAE Level 3. The vehicular control system determines, at least in part via processing of image data captured by the at least one camera, ability of the driver to take over driving the vehicle from the vehicular control system. While the vehicular control system is driving the vehicle in accordance with SAE Level 3, and responsive at least in part to determination by the vehicular control system that the driver is not able to take over driving the vehicle from the vehicular control system, the vehicular control system continues driving the vehicle in accordance with SAE Level 3 and initiates an emergency action.