A method includes forming a bottom source/drain contact plug in a bottom inter-layer dielectric. The bottom source/drain contact plug is electrically coupled to a source/drain region of a transistor. The method further includes forming an inter-layer dielectric overlying the bottom source/drain contact plug. A source/drain contact opening is formed in the inter-layer dielectric, with the bottom source/drain contact plug exposed through the source/drain contact opening. A dielectric spacer layer is formed to have a first portion extending into the source/drain contact opening and a second portion over the inter-layer dielectric. An anisotropic etching is performed on the dielectric spacer layer, and a remaining vertical portion of the dielectric spacer layer forms a source/drain contact spacer. The remaining portion of the source/drain contact opening is filled to form an upper source/drain contact plug.