A method for manufacturing a semiconductor device includes forming a first dummy gate on a substrate, performing a doping process to the substrate, thereby forming a source and a drain at sides of the first dummy gate, performing a first high temperature annealing to activate the source and drain, forming an inter-layer dielectric (ILD) material on the substrate, removing the first dummy gate to create an ILD trench, forming a first high-k dielectric layer within the ILD trench, forming a first dummy cap portion within the ILD trench over the first high-k dielectric layer, performing a second high-temperature annealing to reduce defects in the first high-k dielectric layer, and thereafter, replacing the first dummy cap portion with a first metal gate electrode.