A sound source localization system includes a microphone array composed of a plurality of microphones each converting sound wave into a corresponding voice signal; a room shape estimator that determines a room shape including a location map and a corresponding template voice feature map composed of template voice features associated with a virtual sound source disposed at different locations respectively, and outputs a room reliability indicating confidence about the determined room shape; a lookup table (LUT) that pre-stores the location map and the corresponding template voice feature map; and a localizer that determines a location of a sound source according to the room reliability and similarity between a voice feature associated with the sound source and the template voice features of the template voice feature map.