Embodiments of the present disclosure are directed towards generating images conditioned on a desired attribute. In particular, an attribute-based image generation system can use a directional-GAN architecture to generate images conditioned on a desired attribute. A latent vector and a desired attribute are received. A feature subspace is determined for the latent vector using a latent-attribute linear classifier trained to determine a relationship between the latent vector and the desired attribute. An image is generated using the latent vector such that the image contains the desired attribute. In embodiments, where the feature space differs from a desired feature subspace, a directional vector is applied to the latent vector that shifts the latent vector from the feature subspace to the desired feature subspace. This modified latent vector is then used during generation of the image.