A video encoding system in which pixel data is decomposed into frequency bands prior to encoding. The frequency bands for a slice of a frame may be buffered so that complexity statistics may be calculated across the frequency bands prior to encoding. The statistics may then be used by a rate control component in determining quantization parameters for the frequency bands for modulating the rate in the encoder for the current slice. The quantization parameters for the frequency bands may be calculated jointly to optimize the quality of the displayed frames after decoder reconstruction and wavelet synthesis on a receiving device. Information about one or more previously processed frames may be used in combination with the statistics for a current slice in determining the quantization parameters for the current slice.