A three-dimensional memory device includes source-level material layers located over a substrate, the source-level material layers containing a source contact layer, an alternating stack of insulating layers and electrically conductive layers located over the substrate-level material layers, memory stack structures extending through the alternating stack, such that each of the memory stack structures includes a memory film and a vertical semiconductor channel having a bottom surface that contacts a respective horizontal surface of the source contact layer, and dielectric pillar structures embedded within the substrate-level material layers and located between the memory stack structures.