An image processing method according to the present invention acquires a first image and second image that are picked up through a single image-taking optical system, that are images after a pupil division, and that have a parallax to each other, and then performs a filtering process for each pixel of the first and second images, using first and second transform filters that correspond to the parallax for the pixel and that are of first and second transform filter groups to be respectively applied to the first and second images for the transform into third and fourth images in which the parallax amount and blur amount of the first and second images have been altered.