A system and a method is disclosed for verifying a voice of a user conducting a telephone transaction. The system and method includes a mechanism for prompting the user to speak in a limited vocabulary. A feature extractor converts the limited vocabulary into a plurality of speech frames. A pre-processor is coupled to the feature extractor for processing the plurality of speech frames to produce a plurality of processed frames. The processing includes frame selection, which eliminates each of the plurality of speech frames having an absence of words. A Viterbi decoder is also coupled to said feature extractor for assigning a frame label to each of the plurality of speech frames to produce a plurality of frame labels. The processed frames and frame labels are then combined to produce a voice model, which includes each of the plurality of frame labels that correspond to the number of plurality of processed frames. A mechanism is also provided for comparing the voice model with the claimant's voice model, derived during a previous enrollment session. The voice model also is compared with an alternate voice model set, derived during previous enrollment sessions. The identity claimed is accepted if the voice model matches the claimant's voice model better than the alternative voice model set.