After forming a laterally contacting pair of a semiconductor fin and a conductive strap structure having a base portion vertically contacting a deep trench capacitor embedded in a substrate and a fin portion laterally contacting the semiconductor fin, conducting spikes that are formed on the sidewalls of the deep trench are removed or pushed deeper into the deep trench. Subsequently, a dielectric cap that inhibits epitaxial growth of a semiconductor material thereon is formed over at least a portion of the base portion of the conductive strap structure. The dielectric cap can be formed either over an entirety of the base portion having a stepped structure or on a distal portion of the base portion.