Color handles are generated based on digital image color gradients using machine learning. In order to train a neural network of a generator module of a GAN system, a training digital image is used by the neural network of the generator module to generate a candidate color handle. The discriminator module of the GAN system then compares the candidate color handle with a ground truth color handle of the training digital image using a loss function as part of machine learning, and uses a result of this comparison to train the neural network of the generator module.