A method and system are provided for controlling braking a vehicle in an autonomous driving mode. For instance, the vehicle is controlled in the autonomous driving mode according to a first braking control mode using a first model to adjust the position of a vehicle relative to an expected position of a current trajectory of the vehicle. Using a second model how close to a maximum deviation threshold the vehicle would come if a maximum braking strength for the vehicle was applied is predicted. The maximum deviation threshold provides an allowed forward deviation from the current trajectory. Based on the prediction, the vehicle is controlled in the autonomous driving mode according to a second braking control mode by automatically applying the maximum braking strength.