Techniques and systems are disclosed for context-dependent speech recognition. The techniques and systems described enable accurate recognition of speech by accessing sub-libraries associated with the context of the speech to be recognized. These techniques translate audible input into audio data at a smart device and determine context for the speech, such as location-based, temporal-based, recipient-based, and application based context. The smart device then accesses a context-dependent library to compare the audio data with phrase-associated translation data in one or more sub-libraries of the context-dependent library to determine a match. In this way, the techniques allow access to a large quantity of phrases while reducing incorrect matching of the audio data to translation data caused by organizing the phrases into context-dependent sub-libraries.