A method of forming a semiconductor device that includes forming a high-k gate dielectric layer on a semiconductor substrate, wherein an oxide containing interfacial layer can be present between the high-k gate dielectric layer and the semiconductor substrate. A scavenging metal stack may be formed on the high-k gate dielectric layer. An annealing process may be applied to the scavenging metal stack during which the scavenging metal stack removes oxide material from the oxide containing interfacial layer, wherein the oxide containing interfacial layer is thinned by removing of the oxide material. A gate conductor layer is formed on the high-k gate dielectric layer. The gate conductor layer and the high-k gate dielectric layer are then patterned to provide a gate structure. A source region and a drain region are then formed on opposing sides of the gate structure.