A method of manufacturing a semiconductor device includes forming a gate insulation layer pattern on a substrate, forming a sacrificial layer including impurities on the gate insulation layer pattern, annealing the sacrificial layer so that the impurities in the sacrificial layer diffuse into the gate insulation layer pattern, removing the sacrificial layer, and forming a gate electrode on the gate insulation layer pattern.