There is provided an image processing apparatus including a distortion component calculation unit calculating a distortion component in a time series, based on an image signal indicating a moving image composed of a frame image, for each of a first region corresponding to the whole frame image, a second region corresponding to one region obtained by dividing the frame image, and a third region corresponding to the other region obtained by dividing the frame image, and a decision unit deciding a scanning pattern of an imaging device having captured the moving image, which the image signal indicates, based on a combination of a correlation of distortion between the second region and the first region and a correlation of distortion between the second region and the third region.