A three-dimensional memory device includes an alternating stack of insulating layers and electrically conductive layers, an array of memory structures, conductive structures located between a substrate and the alternating stack, conductive via structures, including an upper portion that overlies and contacts a top surface of a respective one of the electrically conductive layers, and a lower portion that underlies and is adjoined to the upper portion, contacts a top surface of a respective one of the conductive structures, and is electrically insulated from the rest of the electrically conductive layers. Inner, outer and intermediate dielectric spacers laterally surround a respective one of the conductive via structures.