A disclosed method of authenticating a telephone caller includes receiving an authentication request for the telephone caller that includes a voice recording of the caller and household information indicative of a household, creating a first voice pattern based on the voice recording of the caller, retrieving a second voice pattern using at least some of the household information, comparing the two voice patterns, and transmitting information indicative of the results of the comparison. The transmitted information may include information that the telephone caller is authenticated, is not authenticated, or that the comparison results were inconclusive. The method may include monitoring telephone calls with the household, and if it is determined that a voice on at least some of the telephone calls is associated with the household, creating the second voice pattern based on the voice, and associating the second voice pattern to the household. The method may further include disassociating the second voice pattern from the household if it is determined that a change of household has occurred.