A method includes removing a dummy gate stack to form an opening between gate spacers, selectively forming an inhibitor film on sidewalls of the gate spacers, with the sidewalls of the gate spacers facing the opening, and selectively forming a dielectric layer over a surface of a semiconductor region. The inhibitor film inhibits growth of the dielectric layer on the inhibitor film. The method further includes removing the inhibitor film, and forming a replacement gate electrode in a remaining portion of the opening.