A first stack of alternating layers including first electrically conductive layers and first electrically insulating layers is formed with first stepped surfaces and a first dielectric material portion thereupon. Dielectric pillar structures including a dielectric metal oxide can be formed through the first stepped surfaces. Lower memory openings can be formed, and filled with a disposable material or a lower memory opening structure including a lower semiconductor channel and a doped semiconductor region. At least one dielectric material layer and a second stack of alternating layers including second electrically conductive layers and second electrically insulating layers can be sequentially formed. Upper memory openings can be formed through the second stack and the at least one dielectric material layer. A memory film and a semiconductor channel can be formed after removal of the disposable material, or an upper semiconductor channel can be formed on the doped semiconductor region.