A method includes forming a gate dielectric layer and a dummy gate layer; forming a mask over the dummy gate layer; patterning the gate dielectric layer and the dummy gate layer to form a dummy gate structure, the dummy gate structure including a remaining portion of the gate dielectric layer and a remaining portion of the dummy gate layer; epitaxially growing a first spacer layer on the dummy gate structure and the substrate, in which the first spacer layer has a higher growth rate on the exposed surfaces of the dummy gate structure and the substrate than on exposed surfaces of the mask; doping the first spacer layer to form a doped spacer layer having a different lattice constant than the substrate; depositing a second spacer layer over the doped spacer layer; and etching the second spacer layer and the doped spacer layer to form a gate spacer.