The disclosure relates to using ambient lighting conditions with passenger and goods pickups and drop offs with autonomous vehicles. For instance, a map of ambient lighting conditions for stopping locations may be generated by receiving ambient lighting condition data for predetermined stopping locations and arranging this data into a plurality of buckets based on time and one of the stopping locations. A vehicle may then be controlled in an autonomous driving mode in order to stop for a passenger by both observing ambient lighting conditions for different stopping locations and, in some instances, also using the map.