Techniques are disclosed for utilizing passenger attention data captured in vehicles for vehicle localization and location-based services. In an embodiment, a method includes receiving sensor data from one or more sensors of an autonomous vehicle operating in an environment, and generating passenger attention data for a passenger of the autonomous vehicle using the sensor data. A processing circuit determines one or more physical features in the environment based on the passenger attention data. The processing circuit obtains physical feature data associated with the physical feature(s) and initiates one or more actions on the AV based at least in part on the physical feature data. The actions include but are not limited to requesting content for physical features(s) and assisting in localization of the autonomous vehicle.