A spatial decoder unit (23) is arranged for transforming one or more audio channels (s; 1, r) into a pair of binaural output channels (Ib, rb). The device comprises a parameter conversion unit (234) for converting the spatial parameters (sp) into binaural parameters (bp) containing binaural information. The device additionally comprises a spatial synthesis unit (232) for transforming the audio channels (L, R) into a pair of binaural signals (Lb, Rb) while using the binaural parameters (bp). The spatial synthesis unit (232) preferably operates in a transform domain, such as the QMF domain.