A conditional adversarial latent model (CALM) process can be used to generate reference motions from a set of original reference movements to create a library of new movements for an agent. The agent can be a virtual representation various types of characters, animals, or objects. The CALM process can receive a set of reference movements and a requested movement. An encoder can be used to map the requested movement onto a latent space. A low-level policy can be employed to produce a series of latent space joint movements for the agent. A conditional discriminator can be used to provide feedback to the low-level policy to produce stationary distributions over the states of the agent. A high-level policy can be employed to provide a macro movement control over the low-level policy movements, such as providing direction in the environment. The high-level policy can utilize a reward or a finite-state machine function.