World objects tracking and prediction by an autonomous vehicle is disclosed. A method includes receiving, from sensors of the AV, a first observation data; associating the first observation data with a first world object; determining hypotheses for the first world object, wherein a hypothesis corresponds to an intention of the first world object; determining a respective hypothesis likelihood of each of the hypotheses indicating a likelihood that the first world object follows the intention; determining, for at least one hypothesis of the hypotheses, a respective state, wherein the respective state comprises predicted positions of the first world object; and in response to a query, providing a hypothesis of the hypotheses based on the respective hypothesis likelihood of each of the hypotheses.