An apparatus including a transformation circuit and a scaling/quantization circuit. The transformation circuit may be configured to generate one or more transform coefficients in response to a video stream and one or more first control signals. The transformation circuit may be further configured to limit transform coefficients for residual 4×4 blocks to a 16-bit value when the video stream comprises 8-bit video data. The scaling/quantization circuit may be configured to generate one or more quantization coefficients in response to the one or more transform coefficients and one or more second control signals.