Systems and methods for tracking the position of a head-mounted display (HMD) system component, such as a wearable HMD device or a hand-held controller. The HMD component may include a support structure that carries a plurality of angle sensitive detectors that are able to detect the angle of arrival of light emitted from a light source. A processor causes light sources to emit light according a specified pattern, and receive sensor data from the plurality of angle sensitive detectors. The processor may process the received sensor data using machine learning or other techniques to track a position of the head-mounted display component based on the processing of the received sensor data.