Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing images using an image processing neural network system. One of the system includes a shared encoder neural network implemented by one or more computers, wherein the shared encoder neural network is configured to: receive an input image from a target domain; and process the input image to generate a shared feature representation of features of the input image that are shared between images from the target domain and images from a source domain different from the target domain; and a classifier neural network implemented by the one or more computers, wherein the classifier neural network is configured to: receive the shared feature representation; and process the shared feature representation to generate a network output for the input image that characterizes the input image.