This disclosure describes methods, non-transitory computer readable storage media, and systems that utilize image-guided model inversion of an image classifier with a discriminator. The disclosed systems utilize a neural network image classifier to encode features of an initial image and a target image. The disclosed system also reduces a feature distance between the features of the initial image and the features of the target image at a plurality of layers of the neural network image classifier by utilizing a feature distance regularizer. Additionally, the disclosed system reduces a patch difference between image patches of the initial image and image patches of the target image by utilizing a patch-based discriminator with a patch consistency regularizer. The disclosed system then generates a synthesized digital image based on the constrained feature set and constrained image patches of the initial image.