A silicon oxynitride film is formed on entire surface of a semiconductor substrate, a lanthanum oxide film is formed on the silicon oxynitride film and the lanthanum oxide film is removed from a pMOS region. Then, a nitrided hafnium silicate film serving as a highly dielectric film is formed on the entire surface, an aluminum-containing titanium nitride film is formed, a polysilicon film is formed, and the stacked films are patterned into a gate electrode configuration. Next, impurities are introduced into a source/drain region, and an annealing for activating the impurities is utilized to diffuse the aluminum included in the aluminum-containing titanium nitride film to the interface between the silicon oxynitride film and the nitrided hafnium aluminum silicate film in the pMOS region.