The invention discloses a residual echo elimination method and device, wherein the method includes the steps of: carrying out feature extraction on a current received echo signal to obtain a residual echo vector; obtaining a stored user voice template vector, wherein the user voice template vector is obtained by carrying out voice feature extraction on a prerecorded user voice; judging whether the distance between the residual echo vector and the user voice template vector is larger than a preset vector threshold; and if so, determining that the residual echo vector is a non-user signal and eliminating the non-user signal. By adopting the method and the system, linear residual echoes caused by double-end conversation, echo path change, mismatching of filter orders and the like can be processed, and noise signals and other non-linear residual echoes can be processed.