The present disclosure relates to an image artifact removal system that improves digital images by removing complex artifacts caused by image compression. For example, in various implementations, the image artifact removal system builds a generative adversarial network that includes a generator neural network and a discriminator neural network. In addition, the image artifact removal system trains the generator neural network to reduce and eliminate compression artifacts from the image by synthesizing or retouching the compressed digital image. Further, in various implementations, the image artifact removal system utilizes dilated attention residual layers in the generator neural network to accurately remove compression artifacts from digital images of different sizes and/or having different compression ratios.