Techniques are described that can be used to either compress or expand video. Color compression techniques are described that can be used to compress the wide color gamut content into lower color gamut for inclusion in a baseline layer. Color expansion techniques are described that convert lower color gamut data into wider color gamut format for inclusion in an enhancement layer. Both of the baseline video stream and enhancement layer video streams may be transmitted through a channel or stored in a memory device to be viewed later. Accordingly, both baseline and enhancement video layers are available so that either lower or higher quality displays can be used to display video.