Techniques for representing sensor data and predicted behavior of various objects in an environment are described herein. For example, an autonomous vehicle can represent prediction probabilities as an uncertainty model that may be used to detect potential collisions, define a safe operational zone or drivable area, and to make operational decisions in a computationally efficient manner. The uncertainty model may represent a probability that regions within the environment are occupied using a heat map type approach in which various intensities of the heat map represent a likelihood of a corresponding physical region being occupied at a given point in time.