A three-dimensional memory device includes an alternating stack of insulating layers and electrically conductive layers located over a substrate, an array of memory opening fill structures located within an array of memory openings vertically extending through the alternating stack, and a drain-select-level isolation structure vertically extending through drain-select-level electrically conductive layers between two rows of memory opening fill structures. The drain-select-level isolation structure may comprise a low-k dielectric material or an air gap.