Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for implementing a neural network. In one aspect, the neural network includes a batch renormalization layer between a first neural network layer and a second neural network layer. The first neural network layer generates first layer outputs having multiple components. The batch renormalization layer is configured to, during training of the neural network on a current batch of training examples, obtain respective current moving normalization statistics for each of the multiple components and determine respective affine transform parameters for each of the multiple components from the current moving normalization statistics. The batch renormalization layer receives a respective first layer output for each training example in the current batch and applies the affine transform to each component of a normalized layer output to generate a renormalized layer output for the training example.