A camera system processes images based on image luminance data. The camera system includes an image sensor, an image pipeline, an encoder and a memory. The image sensor converts light incident upon the image sensor into raw image data. The image pipeline converts raw image data into color-space image data and calculates luminance levels of the color-space image data. The encoder can determine one or more of quantization levels, determining GOP structure or reference frame spacing for the color-space image data based on the luminance levels. The memory stores the color-space image data and the luminance levels.