A method includes forming a gate stack over a semiconductor region, and forming a first gate spacer on a sidewall of the gate stack. The first gate spacer includes an inner sidewall spacer, and a dummy spacer portion on an outer side of the inner sidewall spacer. The method further includes removing the dummy spacer portion to form a trench, and forming a dielectric layer to seal a portion of the trench as an air gap. The air gap and the inner sidewall spacer in combination form a second gate spacer. A source/drain region is formed to have a portion on an outer side of the second gate spacer.