Aspects of the disclosure relate to controlling a vehicle in an autonomous driving mode. For instance, a first location corresponding to a location where the vehicle is to pick up or drop off a passenger is received. A first cost for the vehicle to reach the first location is determined. A second location based on the first location is identified, and a second cost is determined based on a cost for the vehicle to reach the second location and a cost for the passenger to reach the second location. The first cost is compared to the second cost, and a notification is sent based on the notification. In response to sending the notification, instructions to proceed to the second location are received, and in response to receiving the instructions, the vehicle is controlled in the autonomous driving mode to the second location to pick up or drop off the passenger.