An online system presenting content items to a user generates a model that predicts a latent metric describing user actions that occur at least a reasonable amount of time after presentation of content items. To determine the latent metric, the online system retrieves one or more models predicting likelihoods of the user performing various interactions when presented with the content items and determines weights associated with different retrieved models. Combining the weighted retrieved models generates a model for determining the latent metric. As the retrieved models are based on data accessible to the online system in less than the reasonable amount of time after presenting content items, weighing the retrieved models allows the online system to predict the latent metric describing user actions occurring after content items are presented. When selecting content items for the user, the online system accounts for the latent metric determined by the generated model.