In one embodiment, a computing system may access a training image and a reference image of a person and an incomplete image. A generate may generate an in-painted image based on the incomplete image, and a discriminator may be used to determine whether each of the in-painted image, the training image, and the reference image is likely generated by the generator. The system may compute losses based on the determinations and update the discriminator accordingly. Using the updated discriminator, the system may determine whether a second in-painted image generated by the generator is likely generated by the generator. The system may compute a loss based on the determination and update the generator accordingly. Once training is complete, the generator may be used to generate a modified version of a given image, such as making the eyes of a person appear open even if they were closed in the input image.