A virtual reality (VR) head-mounted display (HMD), a computer-implemented method, and a VR tracking system are described. Generally, a VR HMD includes an inertial measurement unit (IMU) and an optical sensor. When a second VR HMD is located in a same physical environment, the VR HMD can be operated to track a motion of the second VR HMD in the physical environment. For example, image data captured by the VR HMD in addition to inertial data of both VR HMDs are used to determine a three dimensional (3D) physical position of the second VR HMD and to track the 3D physical position over time. Three degrees of freedom (DOF) or six DOF for the motion are derived from the tracking.