Second insulating films of gate insulating films each are composed of a high-permittivity dielectric film having a relative dielectric constant of 8 or more and at least one of the high-permittivity dielectric films constituting the second insulating films is doped with at least one kind of impurity metal ions. The valence number of the impurity metal ions differs by 1 from that of metal ions constituting the high-permittivity dielectric films. Due to this doping, at least one of the density and polarity of charged defects in the high-permittivity dielectric films differs between the second insulating films. The threshold voltage of each MISFET is controlled independently.