A method of manufacturing a semiconductor device includes the steps of forming a plurality of gate electrodes, forming a first insulating film over the plurality of gate electrodes such that the first insulating film is embedded in a space between the plurality of gate electrodes, forming a second insulating film over the first insulating film, forming a third insulating film over the second insulating film, forming a photosensitive pattern over the third insulating film, performing etching using the photosensitive pattern as a mask to form a trench extending through the first to third insulating films and reaching a semiconductor substrate, removing the photosensitive pattern, performing etching using the exposed third insulating film as a mask to extend the trench in the semiconductor substrate, removing the third and second insulating films, and forming a fourth insulating film in the trench and over the first insulating film.