An alternating stack of insulating layers and word-line-level spacer material layers is formed over a substrate. Memory opening fill structures including a respective memory film, a respective word-line-level semiconductor channel portion, a respective word-line-level dielectric core laterally, and a respective sacrificial dielectric material portion are formed through the alternating stack. Drain-select-level material layers are formed over the alternating stack and the memory opening fill structures. Drain-select-level openings are formed through the drain-select-level material layers and over the memory opening fill structures. The sacrificial dielectric material portions are removed selective to the word-line-level semiconductor channel portions underneath the drain-select-level openings. Drain-select-level semiconductor channel portions are formed directly on a respective one of the word-line-level semiconductor channel portions.