Contact via openings are formed through a retro-stepped dielectric material portion in a three-dimensional memory device to underlying etch stop structures. The etch stop structures may include a stepped conductive or semiconductor etch stop plate overlying stepped surfaces in the staircase region. The contact via openings are extended through the etch stop structures. Alternatively, electrically conductive layers, including a topmost dummy electrically conductive layer in the staircase region, may be employed as etch stop structures. In this case, the contact via openings can be extended through the electrically conductive layers. Insulating spacers are formed at peripheral regions of the extended contact via openings. Contact via structures surrounded by the insulating spacers are formed in the extended contact via openings to a respective underlying electrically conductive layer.