A method of encoding a video using constrained soft-decision quantization. The soft-decision quantization includes first performing hard-decision quantization to obtain hard quantized coefficients and, then, obtaining a soft quantized coefficient using a rate-distortion calculation over a search range of quantization levels for a transform domain coefficient, wherein the search range of quantization levels for that transform domain coefficient is constrained within a number of quantization levels of a corresponding hard quantized coefficient. The search range may be based upon a fixed threshold, the coefficient position, the hard quantized coefficient magnitude, a threshold value less accumulated distortion, or other factors, including combinations of these factors. The accumulated distortion may be measured by an L1 norm.