A user profile for a plurality of users may be built for speech recognition purposes and for acting as an agent of the user. In some embodiments, a speech processing device automatically receives an utterance from a user. The utterance may be analyzed using signal processing to identify data associated with the user. The utterance may also be analyzed using speech recognition to identify additional data associated with the user. The identified data may be stored in a profile of the user. Data in the user profile may be used to select an acoustic model and/or a language model for speech recognition or to take actions on behalf of the user.