A method includes forming a dummy gate stack over a semiconductor region, forming gate spacers on opposing sides of the dummy gate stack, forming a source/drain region on a side of the dummy gate stack, forming an inter-layer dielectric over the source/drain region, replacing the dummy gate stack with a replacement gate stack, recessing the replacement gate stack to form a recess between the gate spacers, depositing a liner extending into the recess, depositing a masking layer over the liner and extending into the recess, forming an etching mask covering a portion of the masking layer, and etching the inter-layer dielectric to form a source/drain contact opening. The source/drain region is underlying and exposed to the source/drain contact opening. A source/drain contact plug is formed in the source/drain contact opening. A gate contact plug extends between the gate spacers and electrically connecting to the replacement gate stack.