Disclosed is a technique capable of efficiently storing and retaining characteristic data (a restoration filter or the like) of an optical system used for a restoration process in a storage unit with limited storage capacity in consideration of the degree of image restoration. An image processing device includes a characteristic data storage unit 42 which is capable of storing characteristic data of a plurality of types of optical systems, and a restoration processing unit which subjects source image data to a restoration process using a restoration filter based on a point spread function of an optical system to acquire recovered image data. In case where storing new characteristic data in the characteristic data storage unit, characteristic data which is stored in the characteristic data storage unit 42 is controlled based on a restoration evaluation value which is allocated to characteristic data according to the type of optical system.