A semiconductor structure includes a first alternating stack of first insulating layers and first electrically conductive layers having first stepped surfaces and located over a substrate, a second alternating stack of second insulating layers and second electrically conductive layers having second stepped surfaces, and memory opening fill structures extending through the alternating stacks. A contact via assembly is provided, which includes a first conductive via structure vertically extending from a top surface of one of the first electrically conductive layers through a subset of layers within the second alternating stack and through the second retro-stepped dielectric material portion, an insulating spacer located within an opening through the subset of layers, and a second conductive via structure laterally surrounding the insulating spacer and contacting a second electrically conductive layer.