The disclosure includes a system, method, and tangible memory for generating a simulation. The method may include receiving real-world sensor stream data that describes a real-world data stream that is recorded by onboard sensors in one or more test vehicles, wherein the real-world sensor stream data describes real-world active agents and real-world environmental elements. The method may further include generating three-dimensional (3D) models of active agents and 3D models of environmental elements in a vehicle environment. The method may further include generating a simulation of the vehicle environment that synthesizes the real-world active agents with 3D models of active agents and synthesizes the real-world environmental elements with the 3D models of environmental elements.