Provided are systems and methods for controlling a vehicle based on a map that designed using a factor graph. Because the map is designed using a factor graph, positions of the road can be modified in real-time while operating the vehicle. In one example, the method may include storing a map which is associated with a factor graph of variable nodes representing a plurality of constraints that define positions of lane lines in a road and factor nodes between the variable nodes on the factor graph which define positioning constraints amongst the variable nodes, receiving an indication from the road using a sensor of a vehicle, updating positions of the variable nodes based on the indication and an estimated location of the vehicle within the map, and issue commands capable of controlling a steering operation of the vehicle based on the updated positions of the factor nodes.