An acoustic echo removing apparatus detects space information representing a location of a sound source using a plurality of microphone input signals that are received through a plurality of microphones, and generates an acoustic echo estimation signal from a far-end talker voice signal using an adaptive filter coefficient. The acoustic echo removing apparatus detects a double talk segment using the space information, and determines update of the adaptive filter coefficient according to whether the double talk segment exists.