Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for optimizing a future trajectory of a vehicle. In one aspect, a method comprises obtaining respective initial future trajectories for a vehicle navigating in an environment and for each of the other agents in the vicinity of the vehicle for a future time period; obtaining respective cost functions and linearized dynamic functions for the vehicle and the other agents; performing a backward pass through the time steps starting from the last time step until the current time step to generate a respective optimal agent policy for the vehicle; and generating an optimized future trajectory for the vehicle by performing a forward pass through the time steps starting from the current time step until the last time step to select a respective action generated from the respective optimal agent policy for the vehicle at each time step.