Techniques for performing a simulation using a variable autoencoder conditioned with a diffusion model are described herein. For example, the techniques may include a decoder of a variable autoencoder receiving latent variable data representing an action, intent, or attribute of an object from the diffusion model and outputting occupancy information for an object. The occupancy information can be input into a prediction component associated with a vehicle computing device of a vehicle. The prediction component can output a predicted state for one or more objects proximate the vehicle and perform a simulation between the vehicle and the one or more objects.