In a method of manufacturing a semiconductor device with a shallow trench isolation structure, trenches are formed to extend into a semiconductor substrate. Subsequently, a first insulating film is formed to fill the trenches and to cover a whole surface of the semiconductor substrate, and then a first chemical mechanical polishing (CMP) method is carried out to remove the first insulating film such that the first insulating film is left only in the trenches. Subsequently, a second insulating film is formed to fill the trenches and to cover a whole surface of the semiconductor substrate, and a second CMP method is carried out to remove the second insulating film such that the second insulating film is left only in the trenches.