A monolithic three-dimensional memory device includes a first alternating stack of first insulating layers and first electrically conductive layers located over a top surface of a substrate, an insulating cap layer overlying the first alternating stack, a second alternating stack of second insulating layers and second electrically conductive layers and overlying the insulating cap layer, memory openings extending through the second alternating stack, the insulating cap layer, and the first alternating stack, memory stack structures located within the memory openings, and annular spacers located within the insulating cap layer and laterally surrounding a respective one of the memory stack structures.