Techniques for determining and storing relevant context information for a user input, such as a spoken input, are described. In some embodiments, context information is determined to be relevant on an audio frame basis. Context scores for different types of context data (e.g., prior dialog turn data, user profile data, device information, etc.) are determined for individual audio frames corresponding to a spoken input. Based on the corresponding context scores, the most relevant context is stored in a local context cache. The local context cache is updated as subsequent audio frames, of the user input, are processed. The data stored in the context cache is provided to downstream components to perform tasks such as ASR, NLU and SLU.