A first-tier structure includes a first vertically alternating sequence of first continuous insulating layers and first continuous sacrificial material layers and a first-tier retro-stepped dielectric material portion overlying first stepped surfaces of the first vertically alternating sequence. A second vertically alternating sequence of second continuous insulating layers and second continuous sacrificial material layers is formed over the first-tier structure. A vertically alternating stack of insulating plates and dielectric material is formed over the first-tier retro-stepped dielectric material portion. Alternatively, dielectric pillar structures may be formed in lieu of the vertically alternating stack. After formation of memory stack structures, electrically conductive layers replace portions of the first and second continuous sacrificial material layers. Contact via structures are formed through the vertically alternating stack or the dielectric pillar structures, through the first retro-stepped dielectric material portion, and directly on a first subset of the electrically conductive layers.