Techniques associated with generating and maintaining sparse geographic and map data. In some cases, the system may maintain a factor graph comprising a plurality of nodes. In some cases, the nodes may comprise pose data and sensor data associated with an autonomous vehicle at the geographic position represented by the node. The nodes may be linked based on shared trajectories and shared sensor data.