An HD map system represents landmarks on a high definition map for autonomous vehicle navigation, including describing spatial location of lanes of a road and semantic information about each lane, and along with traffic signs and landmarks. The system generates lane lines designating lanes of roads based on, for example, mapping of camera image pixels with high probability of being on lane lines into a three-dimensional space, and locating/connecting center lines of the lane lines. The system builds a large connected network of lane elements and their connections as a lane element graph. The system also represents traffic signs based on camera images and detection and ranging sensor depth maps. These landmarks are used in building a high definition map that allows autonomous vehicles to safely navigate through their environments.