Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing inputs using a neural network system that includes a whitened neural network layer. One of the methods includes receiving an input activation generated by a layer before the whitened neural network layer in the sequence; processing the received activation in accordance with a set of whitening parameters to generate a whitened activation; processing the whitened activation in accordance with a set of layer parameters to generate an output activation; and providing the output activation as input to a neural network layer after the whitened neural network layer in the sequence.