A number of candidate binaural room impulse responses (BRIRs) are analyzed to select one of them as a selected first BRIR that is to be applied to diffuse audio, and another one as a selected second BRIR that is to be applied to direct audio, of a sound program. A first binaural rendering process is performed on the diffuse audio by applying the selected first BRIR and a first head related transfer function (HRTF) to the diffuse audio. A second binaural rendering process is performed on the direct audio by applying the selected second BRIR and a second HRTF to the direct audio. Results of the two binaural rendering processes are combined to produce headphone driver signals. Other embodiments are also described and claimed.