An alternating stack of insulating layers and spacer material layers is formed over a substrate. Memory stack structures are formed through the alternating stack. A pair of backside trenches and a set of nested trenches are simultaneously formed through the alternating stack. Each trench within the set of nested trenches is spaced from any other trench within the set of nested trenches by at least one patterned remaining portion of the alternating stack having a respective shape of an enclosing wall. The at least one patterned remaining portion of the alternating stack is removed from inside to outside using sequential etch processes. A dielectric pillar structure is formed within the pillar-shaped cavity. The sacrificial material layers are replaced with electrically conductive layers. A through-memory-level conductive via structure is formed through the dielectric pillar structure.