Aspects of the subject technology relate to systems and methods for predicting physical characteristics of a physical environment using a physical characterization model trained based on simulated states of a modeled physical environment. A physical characterization model can be generated based on a plurality of simulated states of a modeled physical environment. Specifically, the physical characterization model can be trained by mapping simulated spatial properties of the modeled physical environment temporally across the plurality of simulated states of the modeled physical environment. Further, input state data describing one or more input states of a physical environment can be received. One or more physical characteristics of the physical environment can be predicted by applying the physical characterization model to the one or more input states of the physical environment.