A method and apparatus for processing voice data of a speech received from a speaker are provided. The method includes extracting a speaker feature vector from the voice data of the speech received from a speaker, generating a speaker feature map by positioning the extracted speaker feature vector at a specific position on a multi-dimensional vector space, forming a plurality of clusters indicating features of voices of a plurality of speakers by grouping at least one speaker feature vector positioned on the speaker feature map, and classifying the plurality of speakers according to the plurality of clusters.