A semiconductor device includes a first gate structure that includes a first interfacial layer, a first gate dielectric layer disposed over the first interfacial layer, and a first gate electrode disposed over the first gate dielectric layer. The semiconductor device also includes a second gate structure that includes a second interfacial layer, a second gate dielectric layer disposed over the second interfacial layer, and a second gate electrode disposed over the second gate dielectric layer. The first interfacial layer contains a different amount of a dipole material than the second interfacial layer.