A system for tracking an object in an ambient environment with respect to a head mounted reference frame may allow the ambient object to be rendered in a virtual display, at a virtual position corresponding to its position in the ambient environment, in response to head movement. The system may detect a position of a head mounted device with respect to a fixed frame of reference in the ambient environment, and may detect an position of the ambient object with respect to the fixed frame of reference in the ambient environment. The system may then translate the detected position of the ambient object to the frame of reference of the head mounted device, or to the head mounted reference frame, to determine a position of the ambient object relative to the head mounted device. This newly determined position may be rendered in the virtual display generated by the head mounted device.