A separation signal generation unit generates a plurality of separation signals which are independent from one another from the mixed signals for one frame which are converted into those in a frequency region. A mask processing unit judges a noise condition of a first separation signal for each frequency bin on the basis of the first separation signal and second separation signals. The mask processing unit further removes a first noise component obtained on the basis of a judgment result on the noise condition from the first separation signal. A noise amount measuring unit measures the amount of noise in the first separation signal. A noise signal selection unit selects a noise signal for each frequency bin on the basis of the amount of noise measured by the noise amount measuring unit. A noise removing unit removes a second noise component from a noise removal signal inputted from the mask processing unit. The noise removing unit outputs the noise removal signal obtained by removing the second noise component as a target signal.