The present disclosure is directed to systems and methods of eye tracking for use in various applications, such as virtual reality or augmented reality applications that include head-mounted display devices. An eye tracking subsystem may be provided that includes a plurality of assemblies that each include a light source, such as an LED, a light detector, such as a silicon photodiode, and a polarizer that is configured to prevent light reflected via specular reflection from reaching the light detectors so that only scattered light is detected by the light detectors. Machine learning or other techniques may be used to track or otherwise determine a user's gaze direction, which may be used by one or more components of an HMD device to improve its functionality in various ways.