Video encoders often produce banding artifacts on areas with smooth gradients and low levels of detail/noise. In this disclosure, a video encoding system identifies the banded areas and adjusts coding parameters accordingly. The video coder may include a pre-coding banding detector and a post-coding banding detector. The pre-coding detector may identify regions in the input picture with smooth gradients that are likely to have banding artifacts after encoding. The post-coding detector may identify regions in the reconstructed picture with visible banding. Usage of pre-coding detector and/or post-coding detector depends on how an encoder operates. In a single-pass encoding or during the first pass of a multi-pass encoding, the pre-coding detection maps are used. During picture re-encoding or during later passes of a multi-pass encoding, the post-coding detector maps are used.