A method of forming a three-dimensional memory device includes forming an alternating stack of insulating layers and sacrificial material layers over a substrate, forming a patterned template structure around memory openings in a drain-select-level above the alternating stack, forming drain-select-level isolation structures in trenches in the patterned template structure, forming memory stack structures in the memory openings extending through the alternating stack, where each of the memory stack structures includes a memory film and a vertical semiconductor channel, replacing the sacrificial material layers with word lines, and separately replacing the patterned template structure with a drain select gate electrode.