After forming a recessed conductive material portion over a deep trench capacitor located in a lower portion of a deep trench embedded in a substrate, a hard mask layer is formed over a top semiconductor layer of the substrate and the recessed conductive material portion such that the hard mask layer completely fills the deep trench. Next, the hard mask layer, the top semiconductor layer and the recessed conductive material portion are patterned to form a laterally contacting pair of a semiconductor fin and a conductive strap structure over the deep trench capacitor as well as a dielectric cap embedded in the deep trench. The dielectric cap vertically contacts a lower portion of the conductive strap structure and laterally surrounds a portion of an upper portion of the conductive strap structure that is not in contact with the semiconductor fin.