Methods, systems, and non-transitory computer readable storage media are disclosed for generating a three-dimensional scene based on a natural language phrase. For example, the disclosed system can analyze a natural language phrase to determine dependencies involving entities and commands in the natural language phrase. The disclosed system can then use the dependencies to generate an entity-command representation of the natural language phrase. Additionally, the disclosed system can generate a semantic scene graph for the natural language phrase from the entity-command representation to indicate contextual relationships of the entities and commands. Furthermore, the disclosed system generates the requested three-dimensional scene by using at least one scene of a plurality of available three-dimensional scenes identified using the semantic scene graph of the natural language phrase.