Techniques to predict object behavior in an environment are discussed herein. For example, such techniques may include determining a trajectory of the object, determining an intent of the trajectory, and sending the trajectory and the intent to a vehicle computing system to control an autonomous vehicle. The vehicle computing system may implement a machine learned model to process data such as sensor data and map data. The machine learned model can associate different intentions of an object in an environment with different trajectories. A vehicle, such as an autonomous vehicle, can be controlled to traverse an environment based on object's intentions and trajectories.