A scanned beam tracking system is included in a virtual retinal display. An infrared light source generates light for scanning the viewer's environment in the direction the viewer is looking. A visible light source generates visible light which is scanned on a viewer's retina to generate a virtual image. A common scanning system is used to scan both the non-visible light and the visible light. The visible light is directed into the viewer's eye. The non-visible light is directed away from the viewer's eye into the environment. Infrared reflectors are positioned in the environment. When the infrared light from the virtual retinal display scans over a reflector the reflector directs the infrared light back toward the virtual retinal display. The current pixel of the scanning cycle when the infrared return light is detected corresponds to the position of the reflector.