Embodiments of the disclosure provide for adapting word pronunciations in a speech recognition system to a user(s) based on expected responses. Some embodiments receive input speech and generate a recognition hypothesis for the input speech. The search algorithm may be informed by a pronunciation dictionary. The recognition hypothesis may comprise a sequence of one or more words. Some embodiments compare the recognition hypothesis with at least one expected response to determine if the recognition hypothesis matches the at least one expected response. Some embodiments generate a phoneme sequence for each word in the recognition hypothesis. Some embodiments after determining that the recognition hypothesis matches the at least one expected response, update a set of phoneme sequences in the pronunciation dictionary associated with at least one word of the recognition hypothesis.