A shot image degraded by an asymmetric degrading filter is corrected, the corrected shot image is modified into an updated image, the updated image is degraded by use of the degradation function obtained at the time of shooting, and the residual between the degraded modified image and the shot image is calculated. Then, with the modified image as a new image to be processed, the image modification and update, the image degradation and the residual calculation are iterated until the calculated residual is smaller than a predetermined threshold value. When the residual is smaller than the predetermined threshold value, the corresponding updated image is used as the restored image.