A method of fabricating a semiconductor device includes forming first and second gate dielectric layers on first and second regions of a semiconductor substrate, respectively, forming a first metal-containing layer on the first and second gate dielectric layers, performing a first annealing process with respect to the first metal-containing layer, removing the first metal-containing layer from the first region, forming a second metal-containing layer on an entire surface of the semiconductor substrate, performing a second annealing process with respect to the second metal-containing layer, forming a gate electrode layer on the second metal-containing layer, and partially removing the gate electrode layer, the second metal-containing layer, the first metal-containing layer, the first gate dielectric layer, and the second gate dielectric layer to form first and second gate patterns on the first and second regions, respectively.