An apparatus and method of improving the quality of a speech codec are provided. In the method, a first energy of a signal decoded by a low-band codec is calculated, and a second energy of a signal decoded by a low-band enhancement mode is calculated. Then, when the first energy is less than a first threshold value or less than a product of the second energy and a second threshold value, a size of the decoded signal is scaled. Accordingly, generation of a quantization error with respect to a silence segment is reduced.