An alternating stack of insulating layers and sacrificial material layers is formed over a substrate. Memory openings and support openings are formed through the alternating stack, and memory opening fill structures and support pillar structures are formed in the memory openings and in the support openings, respectively. Via cavities extending to each of the sacrificial material layers are formed through the alternating stack without forming any stepped surfaces in the alternating stack. The via cavities may be formed in areas that do not overlap with the support pillar structures, or in areas that include at least one support pillar structure. Sacrificial via fill structures are formed in the via cavies, and the sacrificial material layers are replaced with electrically conductive layers. The sacrificial via fill structures are removed, and a combination of a tubular dielectric spacer and a contact via structure can be formed in the via cavities.