Cases in which defects are analyzed in a manufacturing process stage in which a pattern is not formed or in a manufacturing process in which a pattern formed on a lower layer does not appear in the captured image are increasing. However, in these cases, there is a problem of not being able to synthesize a favorable reference image and failing to detect a defect when a periodic pattern cannot be recognized in the pattern. In the present invention, a defect occupation rate, which is the percentage of an image being inspected occupied by a defect region, is found, it is determined whether the defect occupation rate is higher or lower than a threshold, and, in accordance with the determination results, it is determined whether to create, as the reference image, an image comprising pixels having the average luminance value of the luminance values of a plurality of pixels contained in the image being inspected. In particular, when the defect occupation rate is low, an image comprising pixels having the average luminance value of the luminance values of a plurality of pixels contained in the image being inspected is used as the reference image.