World objects tracking and prediction by an autonomous vehicle (AV) is disclosed. A method includes identifying, based on first observation data received from sensors of the AV, an oncoming vehicle; identifying, based on second observation data received from the sensors of the AV, a road object; generating a lane-following hypothesis for the oncoming vehicle, the lane-following hypothesis indicating an intention that the oncoming vehicle remain in a current road lane; computing a lane-following reference driveline for the lane-following hypothesis of the oncoming vehicle; in response to determining that the lane-following reference driveline is blocked by the road object, generating a go-around hypothesis for the oncoming vehicle, and computing a go-around reference driveline for the go-around hypothesis; and providing at least one of a go-around trajectory corresponding to the go-around hypothesis or a lane-following trajectory corresponding to the lane-following hypothesis.