A method of manufacturing a semiconductor device includes providing a substrate having a gate structure, a source region, and a drain region formed thereon, and the gate structure includes a gate insulating layer and a gate electrode. The method also includes forming a first stress layer on the substrate, removing the first stress layer, and forming a second stress layer on the substrate.