A method generates binaural headphone playback signals given multiple audio source signals with associated metadata and a binaural room impulse response (BRIR) database, where the audio source signals can be channel-based, object-based, or a mixture of both signals. The method groups the audio source signals according to positions of the audio sources, divides BRIR into blocks and frames, where the BRIR is divided into a direct block and diffuse blocks, and divides each audio source signal into blocks and frames, wherein the source signal is divided into a current block and previous blocks, and the current block is further divided into the frames. The method further averages, for each of previous frames of the source signals, the divided BRIR identified with the grouping result by downmixing the previous frames of the source signals according to the grouping result, and performs a convolution with the downmixed previous frame.