A method includes forming a gate dielectric layer on a semiconductor fin, and forming a gate electrode over the gate dielectric layer. The gate electrode extends on sidewalls and a top surface of the semiconductor fin. A gate spacer is selectively deposited on a sidewall of the gate electrode. An exposed portion of the gate dielectric layer is free from a same material for forming the gate spacer deposited thereon. The method further includes etching the gate dielectric layer using the gate spacer as an etching mask to expose a portion of the semiconductor fin, and forming an epitaxy semiconductor region based on the semiconductor fin.