The present disclosure relates to the design of a fast binaural rendering for multiple moving audio sources. This disclosure takes the audio source signals which can be object-based, channel-based or a mixture of both, associated metadata, user head tracking data and binaural room impulse response (BRIR) database to generate the headphone playback signals. The present disclosure applies a frame-by-frame binaural rendering module which takes parameterized components of BRIRs for rendering moving sources. In addition, the present disclosure applies hierarchical source clustering and downmixing in the rendering process to reduce computational complexity.