A method of manufacturing a semiconductor device comprises the steps of forming a first film and a second film on a semiconductor substrate, selectively removing the second film, the first film and a top portion of the semiconductor substrate to form a first groove, burying a first insulator film in the first groove to form an isolation region, patterning the second film surrounded by the isolation region to form a dummy gate layer, doping the semiconductor substrate with an impurity using the dummy gate layer as a mask, forming a second insulator film on the semiconductor substrate surrounded by the dummy gate layer and the first insulator film, removing the dummy gate layer and the first film to form a second groove, forming a gate insulator film on the semiconductor substrate in the second groove, and forming a gate electrode on the gate insulator film in the second groove.