Methods, computer systems, and apparatus, including computer programs encoded on computer storage media, for generating a prediction that characterizes an environment. The system obtains an input including data characterizing observed trajectories one or more agents and data characterizing one or more map features identified in a map of the environment. The system generates, from the input, an encoder input that comprises representations for each of a plurality of points in a top-down representation of the environment. The system processes the encoder input using a point cloud encoder neural network to generate a global feature map of the environment, and processes a prediction input including the global feature map using a predictor neural network to generate a prediction output characterizing the environment.