A three-dimensional memory device includes an alternating stack of insulating layers and electrically conductive layers having stepped surfaces, memory stack structures extending through the alternating stack, a retro-stepped dielectric material portion overlying the stepped surfaces, and pillar-shaped contact-opening assemblies located within a respective pillar-shaped volume vertically extending through the retro-stepped dielectric material portion and a region of the alternating stack that underlies the retro-stepped dielectric material portion. Some of the pillar-shaped contact-opening assemblies can include a first conductive plug that laterally contacts a cylindrical sidewall of a respective one of the electrically conductive layers and a conductive via structure that contacts a top surface of the first conductive plug.