A method of masking a residual echo signal by an echo canceller is provided. The method comprises receiving a far-end signal, adjusting filter coefficients of an adaptive filter in response to the far-end signal, generating an echo model signal based on the far-end signal using the adaptive filter, receiving a near-end signal, subtracting the echo model signal from the near-end signal to generate an output signal, defining a spectral mask based on the near-end signal, wherein the spectral mask is indicative of near-end spectral peaks and near-end spectral valleys, de-emphasizing the output signal in spectral regions of the near-end spectral peaks, and emphasizing the output signal in spectral regions of the near-end spectral valleys, wherein the de-emphasizing occurs during filter coefficients determination for the adaptive filter. A weighted filter may perform the de-emphasizing and the emphasizing operations, where the weighted filter uses medium term spectral characteristics of the near-end signal.