An image processing server enhances an original image using image data from other images. The server generates global descriptors of a set of received images and creates an image data structure therefrom. The server also identifies a set of patches for each of the received images, generates descriptors of the patches, and generates a patch data structure therefrom. The image processing system identifies a patch in an original image for which enhancement is desired. The image processing server uses a global descriptor of the original image to search the image data structure for a set of related images that are visually similar to the original image. The image processing server uses a patch descriptor of the patch for which enhancement is desired to search the patch data structures for the related images to identify a set of patches that are visually similar to the patch in the original image.