The technology relates to actively looking for an assigned passenger prior to a vehicle 100 reaching a pickup location. For instance, information identifying the pickup location and client device information for authenticating the assigned passenger is received. Sensor data is received from a perception system of the vehicle identifying objects in an environment of the vehicle. When the vehicle is within a predetermined distance from the pickup location, authenticating a client device using the client device information is attempted. When the client device has been authenticated, the sensor data is used to determine whether a pedestrian is within a first threshold distance of the vehicle. When a pedestrian is determined to be within the first threshold distance of the vehicle, the vehicle is stopped prior to reaching the pickup location, to wait for the pedestrian within the first threshold distance of the vehicle to enter the vehicle.