An audio encoder for encoding an audio signal, comprises: a first encoding processor (600) for encoding a first audio signal portion in a frequency domain, wherein the first encoding processor (600) comprises: a time frequency converter for converting the first audio signal portion into a frequency domain representation having spectral lines up to a maximum frequency of the first audio signal portion; a spectral encoder for encoding the frequency domain representation; a second encoding processor for encoding a second different audio signal portion in the time domain; a cross-processor (700) for calculating, from the encoded spectral representation of the first audio signal portion, initialization data of the second encoding processor (610), so that the second encoding processing (610) is initialized to encode the second audio signal portion immediately following the first audio signal portion in time in the audio signal; a controller configured for analyzing the audio signal and for determining, which portion of the audio signal is the first audio signal portion encoded in the frequency domain and which portion of the audio signal is the second audio signal portion encoded in the time domain; and an encoded signal former for forming an encoded audio signal comprising a first encoded signal portion for the first audio signal portion and a second encoded signal portion for the second audio signal portion.