A method includes forming a dummy gate electrode on a semiconductor region, forming a first gate spacer on a sidewall of the dummy gate electrode, and removing an upper portion of the first gate spacer to form a recess, wherein a lower portion of the first gate spacer remains, filling the recess with a second gate spacer, removing the dummy gate electrode to form a trench, and forming a replacement gate electrode in the trench.