The present invention discloses a contact center with speaker identification and verification (SIV) capabilities. In the invention, a set of contact center components can provide automated interactive communications with callers, can provide queue management for callers waiting to communicate with live agents, and can provide skills based routing for assigning live agents to callers. The SIV component can analyze speech utterances to determine a speaker identify based upon biometric characteristics of the analyzed speech utterances. Additionally, the SIV component can process speech from contact center sessions. In one embodiment, the SIV component can prevent agent substitutions from occurring of which the call center is unaware. The SIV component can also be used to distinguish whether communication session content was spoken by a contact center agent or a caller.