Enhanced dynamic range requires more than 8 bit representation for single color components of pixels. For this purpose, normal color resolution images and high color resolution images are available. Backward compatibility can be achieved by a layered approach using a color enhancement layer, and a conventional image as color base layer. Both have same spatial and temporal resolution. Encoding of the color enhancement layer uses prediction and residual. A methods for optimized color enhancement prediction is disclosed. Color bit depth prediction is done by constructing a polynomial that approximates for all pixels of one color component of a block the color enhancement layer from the color base layer. A predicted version of the high color resolution image and a residual are generated and updated by a residual. The coefficients are compressed and added as metadata to the data stream.