Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for determining the likelihood that a particular event would occur during a navigation interaction using simulations generated by sampling from agent data. In one aspect, a method comprises: identifying an instance of a navigation interaction that includes an autonomous vehicle and agents navigating in an environment; generating multiple simulated interactions corresponding to the instance, comprising, for each simulated interaction: identifying one or more agents; for each identified agent and for each property that characterizes behavior of the identified agent, obtaining a probability distribution for the property; sampling a respective value from each of the probability distributions; and simulating the navigation interaction in accordance with the sampled values; and determining a likelihood that the particular event would occur during the navigation interaction based on whether the particular event occurred during each of the simulated interactions.