An apparatus for inspecting pattern defects, the apparatus including: an image acquisition unit which acquires an image of a specimen and stores the acquired image in an image memory; a defect candidate extraction unit which performs a defect candidate extraction process by using the acquired image, which is read from the image memory; and a defect detection unit which performs a defect detection process and a defect classification process based on a partial image containing a defect candidate that is extracted by the defect candidate extraction unit, wherein the processes performed by the defect detection unit is performed off-line asynchronously with an image acquisition process that is performed by the image acquisition unit.