Techniques and tools are described for scalable video encoding and decoding. In some embodiments, an input frame is downsampled in terms of sample depth and chroma sampling rate, encoded, and output from the encoder as a base layer bitstream. The base layer bitstream is also reconstructed and upsampled to produce a reconstructed bitstream which is subtracted from the original input frame to produce a residual layer. The residual layer is split and encoded as a sample depth residual layer bitstream and a chroma high-pass residual layer bitstream. To recover the encoded input frame, a decoder receives one or more of these bitstreams, decodes them, and combines them to form a reconstructed image. The use of separate codecs is allowed for the base layer and the enhancement layers, without inter-layer dependencies.