For purpose of providing a defect inspecting method and an apparatus thereof and a defect inspecting method on basis of electron beam image and an apparatus thereof, reducing possibility of bringing erroneous or false reports due to the test objection side and the inspecting apparatus side, being caused by discrepancies, such as the minute difference in pattern shapes, the difference in gradation values, the distortion or deformation of the patterns, the position shift, thereby enabling the detection of the defects or the defective candidates in more details, wherein an image which is small in distortion by controlling the electron beam scanning is detected and divided into a size so as to be able to neglect therefrom, and then position shift detection and defect decision are carried out in an accuracy less or finer than pixel for each division unit. In the defect decision, a desired tolerance can be set up depending upon changes in gradation values and the position shift.