Higher Order Ambisonics (HOA) signals can be compressed by decomposition into a predominant sound component and a residual ambient component. The compressed representation comprises pre-dominant sound signals, coefficient sequences of the ambient component and side information. For efficiently combining HOA decompression and HOA rendering to obtain loudspeaker signals, combined rendering and decoding of the compressed HOA signal comprises perceptually decoding the perceptually coded portion and decoding the side information, without reconstructing HOA coefficient sequences. For reconstructing components of a first type, fading of coefficient sequences is not required, while for components of a second type fading is required. For each second type component, different linear operations are determined: one for coefficient sequences that in a current frame require no fading, one for those that require fading-in, and one for those that require fading-out. From the perceptually decoded signals of each second type component, faded-in and faded-out versions are generated, to which the respective linear operations are applied.