The invention provides an operation method of a semiconductor system, which includes providing a system which includes a layout pattern to scanning electron microscope (SEM) pattern prediction model (LS model) and a novelty detection model (ND model), inputting a layout pattern to the ND model, and the ND model judges whether the layout pattern is a novel layout pattern, and if the layout pattern is confirmed as the novel layout pattern after judgment, performing a process step on the novel layout pattern to form an SEM (scanning electron microscope) pattern.