One method disclosed includes determining a map of markers fixed in an environment, where the map of markers includes a location and an orientation of each marker. The method further includes determining locations of a set of detected markers relative to the map based on a location of a robotic device relative to the map and based on sensor data from the robotic device. The method also includes associating a detected marker from the set of detected markers with a mapped marker based on the determined location of the detected marker relative to the map and based on a visibility constraint related to the orientation of the mapped marker. The method additionally includes adjusting, in the map, the orientation of the mapped marker based on the determined location of the detected marker relative to the map.