Automated capture of image data for points of interest may be implemented for points of interest in an environment external to a vehicle. Sensors implemented as part of a vehicle may collect sensor data for an environment. Processing of the sensor data may be performed to detect points of interest in the environment. In response to detecting a point of interest, image data may be captured by one or more of the sensors implemented at the vehicle. Different types of image data may be captured, such as panoramic images and three-dimensional reconstructions of a scene. Metadata may be generated for captured image data which may describe the point of interest that is captured by the image data. The image data and the metadata may be stored locally at the vehicle or to a remote data store. The image data may also be shared with other computing devices.