An apparatus for processing a defect candidate image, including: an imager for taking an enlarged image of a specimen; an image processor for processing the image taken by the imager to detect defect candidates existing on the specimen and classify the detected defect candidates into one of plural defect classes; a memory for storing information of the defect candidates including the images of the defect candidates and the classified defect class data outputted from the image processor; and a display unit having a display screen for displaying information stored in the memory, wherein the display unit displays an image of the defect candidates together with the defect class data stored in the memory and the displayed defect class data is changeable on the display screen, and the memory changes the stored defect class data of the displayed defect candidate to the changed defect class data.