This technology relates to optimizing location and orientation information of an image using known locations of places captured within the image. For example, an image and associated pose data including the image's orientation and location may be received. One or more places captured within the image may be determined, with each place having a respective known location. The image may be annotated with the one or more places. A difference between each annotation and its respective known location to obtain updated pose data of the image may be minimized and the associated pose data may be updated to the updated pose data.