A headphone down mix signal (314) can be efficiently derived from a parametric down mix of a multi-channel signal (312), when modified HRTFs (310) (head related transfer functions) are derived from HRTFs (308) of a multi-channel signal using a level parameter (306) having information on a level relation between two channels of the multi-channel signals such that a modified HRTF (310) is stronger influenced by the HRTF (308) of a channel having a higher level than by the HRTF (308) of a channel having a lower level. Modified HRTFs (310) are derived within the decoding process taking into account the relative strength of the channels associated to the HRTFs (308). The HRTFs (308) are thus modified such that a down mix signal (314) of a parametric representation of a multi-channel signal can directly be used to synthesize the headphone down mix signal (314) without the need of an intermediate full parametric multi-channel reconstruction of the parametric down mix.