In a method of inspecting a defect and an apparatus thereof, an allowable range for a gradation value of a difference image is determined for each pixel from one pixel or less of position shift quantity between two images to be compared, a variation rate in a local gradation value of an image, and a representative value of the local gradation value. Then, by comparing the gradation value of the difference image with the allowable range determined for each pixel, a pixel, on which the gradation value of the difference image is within the allowable range, is judged to be an non-defective candidate and a pixel, on which the gradation value of the difference image is beyond the allowable range, is judged to be a defective candidate. This reduces the number of false information caused by mismatches attributed to an object to be inspected and an image detecting system, such as an infinitesimal difference in pattern configuration, a difference in the gradation value, a distortion of a pattern, and a position shift, thus making it possible to detect a more minute defect.