A three-dimensional memory device includes an alternating stack of insulating layers and electrically conductive layers located over a substrate, memory openings located in a memory array region and vertically extending through the alternating stack, memory opening fill structures located in the memory openings, and laterally-isolated contact via assemblies located in a contact region. Each of the laterally-isolated contact via assemblies includes a contact via structure contacting a top surface of a respective one of the electrically conductive layers and an insulating spacer laterally surrounding the contact via structure and having an outer surface having a corrugated vertical cross-sectional profile in which first portions of the insulating spacer located at levels of the electrically conductive layers laterally protrude outward relative to second portions of the insulating spacer located at levels of the insulating layers.