A three-dimensional memory device includes an alternating stack of insulating layers and electrically conductive layers having stepped surfaces in a contact region, memory openings vertically extending through the alternating stack, memory opening fill structures located in the memory openings, at least one retro-stepped dielectric material portion overlying the alternating stack, finned dielectric pillar structures vertically extending through the alternating stack in the contact region, support pillar structures, and layer contact via structures vertically extending through the at least one retro-stepped dielectric material portion. Each of the layer contact via structures contacts a respective one of the electrically conductive layers and a respective one of the finned dielectric pillar structures.