Various implementations include devices, systems and approaches for tracking user orientation. In some cases, a wearable audio device includes: an electro-acoustic transducer for providing an audio output; an inertial measurement unit (IMU) for indicating motion of the wearable audio device; a set of distance sensors; and a controller coupled with the electro-acoustic transducer, the IMU and the set of distance sensors, the controller configured to: determine an orientation of a user of the wearable audio device based on the indication of motion from the IMU; verify the orientation of the user as detected by the IMU, using data from the set of distance sensors; and execute an orientation-based action in response to verifying the orientation of the user.