The subject disclosure relates to techniques for increasing a quality of predicted trajectories output from a trained prediction algorithm. A process of the disclosed technology can include receiving by a trained prediction algorithm, information about objects in an environment as perceived by sensors of an autonomous vehicle, receiving by the trained prediction algorithm, information about a location of the autonomous vehicle in the environment, generating by the trained prediction algorithm, a predicted trajectory for an object among the objects in the environment, wherein the predicted trajectory being anchored by a path through a tree of paths including at least two modes, wherein each mode creates a node in the tree, wherein a mode is a semantic classification of a portion of the predicted trajectory, and outputting by the trained prediction algorithm, the predicted trajectory for the object.