The invention relates to a speaker segmented clustering method and system based on factor analysis. The speaker segmented clustering method based on factor analysis comprises steps of 1) extracting acoustic characteristics of training voice, training a gauss mixed general background model, training a total changing factor model and a gauss probability linear determination analysis model, (2) performing segmentation on testing voice and extracting acoustic characteristics of voice segments, (3) mapping the extracted voice characteristics as total changing amount factors according to the gauss mixed general background model and the total changing factor model, loading a gauss probability linear determination analysis model, calculating a log-likelihood ratio score between any two voice segments according to the total changing amount factors, (4) choosing two kinds of segments having highest scores and performing combination, performing gradual iteration until convergence according to a level clustering method and finally outputting a segmented clustering result of the speaker. The speaker segmented clustering method and system based on factor analysis introduce uncertainty of the total changing factor into the gauss probability linear determination model to perform training and marking, and can improve system performance of analyzing based factors on a short-time voice segment.