Implementations are related to observing user interactions in association with searching for various files, and modifying a model and/or index based on such observations in order to improve the search process. In some implementations, a reinforcement learning model is utilized to adapt one or more search actions of the search process. Such search action(s) can include, for example, updating an index, reweighting terms in an index, modifying a search query, and/or modifying one or more ranking signal(s) utilized in raking search results. A policy of the reinforcement learning model can be utilized to generate action parameters that dictate performance of search action(s) for a search query, dependent on an observed state that is based on the search query. The policy can be iteratively updated in view of a reward function, and observed user interactions across multiple search sessions, to generate a learned policy that reduces duration of search sessions.