Methods and systems for detecting defects in patterns formed on a specimen are provided. One system includes one or more components executed by one or more computer subsystems, and the component(s) include first and second learning based models. The first learning based model generates simulated contours for the patterns based on a design for the specimen, and the simulated contours are expected contours of a defect free version of the patterns in images of the specimen generated by an imaging subsystem. The second learning based model is configured for generating actual contours for the patterns in at least one acquired image of the patterns formed on the specimen. The computer subsystem(s) are configured for comparing the actual contours to the simulated contours and detecting defects in the patterns formed on the specimen based on results of the comparing.