There are provided system and method of classifying defects in a semiconductor specimen. The method comprises: upon obtaining by a computer a Deep Neural Network (DNN) trained to provide classification-related attributes enabling minimal defect classification error, processing a fabrication process (FP) sample using the obtained trained DNN; and, resulting from the processing, obtaining by the computer classification-related attributes characterizing the at least one defect to be classified, thereby enabling automated classification, in accordance with the obtained classification-related attributes, of the at least one defect presented in the FP image. The DNN is trained using a classification training set comprising a plurality of first training samples and ground truth data associated therewith, each first training sample comprising a training image presenting at least one defect and the ground truth data is informative of classes and/or class distribution of defects presented in the respective first training samples; the FP sample comprises a FP image presenting at least one defect to be classified.