An apparatus for decoding an encoded audio signal (100) to obtain modified output signals (160), comprises an input interface (110) for receiving a transmitted downmix signal (112) and parametric data (114) relating to audio objects included in the transmitted downmix signal (112), the downmix signal being different from an encoder downmix signal, to which the parametric data is related; a downmix modifier (116) for modifying the transmitted downmix signal using a downmix modification function, wherein the downmix modification is performed in such a way that a modified downmix signal is identical to the encoder downmix signal or is more similar to the encoder downmix signal compared to the transmitted downmix signal (112); an object renderer (118) for rendering the audio objects using the modified downmix signal and the parametric data to obtain output signals; and an output signal modifier (120) for modifying the output signals using an output signal modification function, wherein the output signal modification function is such that a manipulation operation applied to the encoded downmix signal to obtain the transmitted downmix signal (112) is at least partly applied to the output signals to obtain the modified output signals (160).