Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for jointly training an encoder neural network and a decoder neural network. In one aspect, a method comprises, for each latent dimension in a proper subset of a plurality of latent dimensions of a latent space: processing a predefined embedding that represents the latent dimension using the decoder neural network to generate multi-modal data, having a plurality of feature dimensions, that defines a predicted multi-modal data archetype corresponding to the latent dimension; and updating the values of the set of decoder parameters using gradients of an archetype loss function that measures an error between: (i) a predicted multi-modal data archetype corresponding to the latent dimension, and (ii) a target multi-modal data archetype corresponding to the latent dimension.