Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, that determine yield behavior for an autonomous vehicle. An agent that is in a vicinity of an autonomous vehicle can be identified. An obtained crossing intent prediction characterizes a predicted likelihood that the agent intends to cross a roadway during a future time period. First features of the agent and of the autonomous vehicle are obtained. An input that includes the first features and the crossing intent prediction is processed using a machine learning model to generate an intent yielding score that represents a likelihood that the autonomous vehicle should perform a yielding behavior due to the intent of the agent to cross the roadway. From at least the intent yielding score, an intent yield behavior signal is determined and indicates whether the autonomous vehicle should perform the yielding behavior prior to reaching the first crossing region.