An image processing device includes a statistical information acquiring unit, an optical information acquiring unit, a filter information calculating unit and a filter coefficient calculating unit. The filter information calculating unit obtains filter information of a restoration filter for point image restoration processing according to at least one of statistical information and optical information. The filter information includes information related to number of taps of the restoration filter and information indicating a kind of symmetry of the restoration filter. The filter coefficient calculating unit calculates a filter coefficient of the restoration filter according to the statistical information and the optical information with at least the information related to the number of taps of the restoration filter and the information indicating the kind of the symmetry of the restoration filter of the filter information as a constraint condition.