A more efficient way of addressing different bit-depths, or different bit-depths and chroma sampling format requirements is achieved by using a low bit-depth and/or low-chroma resolution representation for providing a respective base layer data stream representing this low bit-depth and/or low-chroma resolution representation as well as for providing a higher bit-depth and/or higher chroma resolution representation so that a respective prediction residual may be encoded in order to obtain a higher bit-depth and/or higher chroma resolution representation. By this measure, an encoder is enabled to store a base-quality representation of a picture or a video sequence, which can be decoded by any legacy decoder or video decoder, together with an enhancement signal for higher bit-depth and/or reduced chroma sub-sampling, which may be ignored by legacy decoders or video decoders.