This application discloses a computing system to implement map building in an assisted or automated driving system. The computing system can track movement of a vehicle based on sensor measurement data populated in an environmental model and vehicle movement measurements. The computing system can correlate the tracked movement of the vehicle to map data based on a previously detected location of the vehicle relative to the map data. The computing system can modify the map data to include the sensor measurement data utilized to track the movement of the vehicle based on the correlation of the tracked movement of the vehicle to map data. The computing system can modify the map data by building a map of the sensor measurement data and the tracked movement of the vehicle, and populating the map data with the built map.