Methods, systems, and apparatus, including computer programs encoded on computer storage media, for updating data of a map. One of the methods includes storing a map of an environment, the map comprising surfels and a road graph; receiving new surfel data for the surfels; adjusting the surfels based on the new surfel data; determining a vector field difference between the surfels of the stored map and the adjusted surfels; adjusting a portion of the road graph based on the vector field difference; generating an updated map comprising the adjusted surfels and the adjusted portion of the road graph; and storing the updated map.