A braking control system for a vehicle includes a camera configured to be disposed at a vehicle so as to have an exterior field of view. After actuation of the brake system by an automatic emergency braking system of the vehicle, a control controls the brake system of the subject vehicle. While the control is controlling the brake system of the subject vehicle, the braking control system determines if the driver of the subject vehicle is impaired. Responsive to a determination that the driver of the subject vehicle is impaired, the braking control system does not allow the driver to override the control's control of the brake system of the subject vehicle, and responsive to a determination that the driver of the subject vehicle is not impaired, the braking control system allows the driver to override the control's control of the brake system of the subject vehicle.