Aspects of the disclosure relate to controlling a vehicle in an autonomous driving mode. The system includes a plurality of sensors configured to generate sensor data. The system also includes a first computing system configured to generate trajectories using the sensor data and send the generated trajectories to a second computing system. The second computing system is configured to cause the vehicle to follow a receive trajectory. The system also includes a third computing system configured to, when there is a failure of the first computer system, generate and send trajectories to the second computing system based on whether a vehicle is located on a highway or a surface street.