The present invention is to allow rapid detection of such a defect as buried in a pixel positional deviation, expansion/contraction noise or sensing noise on an image. A relationship between an inspection reference pattern image and a pattern image to be inspected is identified during inspection to construct a mathematical model obtained by absorbing (applying fitting on) a pixel positional deviation, expansion/contraction noise or sensing noise on an image, and a defect is detected by comparing a new inspection reference pattern image (model image) obtained by simulating the mathematical model and a pattern image to be inspected.