A memory opening can be formed through a multiple tier structure. Each tier structure includes an alternating stack of sacrificial material layers and insulating layers. After formation of a dielectric oxide layer, the memory opening is filled with a sacrificial memory opening fill structure. The sacrificial material layers are removed selective to the insulating layers and the dielectric oxide layer to form backside recesses. Physically exposed portions of the dielectric oxide layer are removed. A backside blocking dielectric and electrically conductive layers are formed in the backside recesses. Subsequently, the sacrificial memory opening fill structure is replaced with a memory stack structure including a plurality of charge storage regions and a semiconductor channel. Hydrogen or deuterium from a dielectric core may then be outdiffused into the semiconductor channel.