A system and method provide for determining regions of interest within an image based on viewer interaction with the image. At least one image associated with a location is provided for display in a viewport, and pose data related to user interaction with the at least one image is identified. Weights are assigned to portions of the at least one image based on the pose data, the weights indicating at least a period of time the portion of the at least one image is generally at a center of the viewport. Based on the assigned weights, image regions of interest of the at least one image are determined.