Provided is a method of deconvoluting and restoring an image observed in a charged particle beam apparatus. The method includes receiving, by an image processing apparatus, an observed image acquired by a detector of the charged particle beam apparatus, calculating, by the image processing apparatus, a point spread function (PSF), deconvoluting and restoring, by the image processing apparatus, the observed image using the observed image and the PSF, calculating, by the image processing apparatus, an evaluation function of the parameter applied to a process of the deconvoluting, and adjusting, by the image processing apparatus, the parameter on the basis of a result of the evaluation function, and restoring the image after deconvoluting the observed image and the PSF again using an optimal parameter. Furthermore, a charged particle beam apparatus using the above-described method of deconvoluting and restoring the image is provided.