In a method of manufacturing a semiconductor device having a memory mat portion in which an active region and a field region are formed densely, after a polishing stopper film is deposited on a semiconductor substrate, there are formed grooves by etching a polishing stopper film of a field region and the semiconductor substrate. Then, after an insulating film is deposited so as to fill the grooves, then insulating film is partly removed from the memory mat portion by etching. Under this state, the insulating film is chemically mechanically polished until the polishing stopper film is exposed. The film thickness of the polishing stopper film on the active region can be reduced, and an electrical element isolation characteristic of the field region can be improved. At the same time, upon chemical mechanical polishing, a silicon substrate can be prevented from being exposed at the central portion of the memory mat portion and the insulating film can be prevented from being left on the silicon nitride film near the outer periphery, thereby making it possible to form elements having uniform electrical characteristics on all active regions of the memory mat portion.