A three-dimensional memory device includes an alternating stack of insulating layers and electrically conductive layers located over at least one source layer, and groups of memory opening fill structures vertically extending through the alternating stack. Each memory opening fill structure can include a vertical stack of memory elements and a vertical semiconductor channel. A plurality of source-side select gate electrodes can be laterally spaced apart by source-select-level dielectric isolation structures. Alternatively or additionally, the at least one source layer may include a plurality of source layers. A group of memory opening fill structures can be selected by selecting a source layer and/or by selecting a source-level electrically conductive layer.