A method of generating binaural headphone playback signals given multiple audio source signals with an associated metadata and binaural room impulse response (BRIR) database, wherein the multiple audio source signals can be channel-based, object-based, or a mixture of both signals. The method includes grouping the multiple audio source signals according to positions of the audio sources in a hierarchical manner, and parameterizing BRIR to be used for rendering. The method also includes dividing each audio source signal to be rendered into a number of blocks and frames, averaging the parameterized BRIR sequences identified with a hierarchically grouping result, and downmixing the divided audio source signals identified with the hierarchically grouping result.