An image processing apparatus includes a first memory used for first rearrangement processing on a group of pixels in an input image, and a second memory used for second rearrangement processing on a group of pixels in an image obtained by the first rearrangement processing, and performs correction processing that includes the first rearrangement processing and the second rearrangement processing on the input image. One of the first and second memories is capable of higher-speed random access than the other memory and has a smaller memory capacity than the other memory. One of the first rearrangement processing and the second rearrangement processing is processing for rearranging a group of pixels in each of a plurality of block images generated from the input image, and the other rearrangement processing is processing for rearranging pixel rows among the block images. The one rearrangement processing involves random access to the one memory.