An alternating stack of insulating layers and sacrificial material layers is formed over a substrate. Memory openings are formed through the alternating stack. A layer stack including a charge storage layer, a tunneling dielectric layer, a semiconductor material layer, and a dielectric material layer is formed in the memory openings. The dielectric material layer may include a doped silicate glass layer. A doped silicate glass pillar can be formed at a bottom portion of each memory opening, and a bottom portion of the semiconductor material layer can be converted into a source region by outdiffusion of dopants from the doped silicate glass pillar. Alternatively, the semiconductor material layer can be heavily doped, and can be recessed to form a source region.