During an initialization of a head pose tracker for a spatial audio system, a spatial audio ambience bed is rotated about a boresight vector to align the boresight vector with a center channel of the ambience bed. The boresight is computed using source device motion data and headset motion data. The ambience bed includes the center channel and one or more other channels. An ambience bed reference frame is aligned with a horizontal plane of a headset reference frame, such that the ambience bed is horizontally level with a user's ears. A first estimated gravity direction is fixed (made constant) in the ambience bed reference frame. During head pose tracking, the ambience bed reference frame is rolled about the boresight vector to align a second estimated gravity direction in the headset reference frame with the first estimated gravity direction fixed in the ambience bed reference frame.