A sound source separation device includes a first microphone that picks up a first voice, a second microphone that picks up a second voice, a first crosstalk canceller that removes, from a voice signal of the first microphone, first crosstalk caused when the second voice is picked up by the first microphone, and a second crosstalk canceller that removes, from a voice signal of the second microphone, second crosstalk caused when the first voice is picked up by the second microphone. The first crosstalk canceller uses a voice signal in which the second crosstalk is removed from the voice signal of the second microphone to estimate and calculate a first interference signal indicative of a degree of the first crosstalk, and to remove the calculated first interference signal from the voice signal of the first microphone. The second crosstalk canceller uses a voice signal in which the first crosstalk is removed from the voice signal of the first microphone to estimate and calculate a second interference signal indicative of a degree of the second crosstalk, and to remove the calculated second interference signal from the voice signal of the second microphone.