Sensor data identifying an emergency vehicle approaching the autonomous vehicle may be received. A predicted trajectory for the emergency vehicle may be received. Whether the autonomous vehicle is impeding the emergency vehicle may be determined based on the predicted trajectory and map information identifying a road on which the autonomous vehicle is currently traveling. Based on a determination that the autonomous vehicle is impeding the emergency vehicle, the autonomous vehicle may be controlled in an autonomous driving mode in order to respond to the emergency vehicle.