A three-dimensional memory device includes an alternating stack of insulating layers and electrically conductive layers containing steps, memory stack structures extending through the alternating stack, a first contact via structure which contacts a top surface of a respective upper electrically conductive layer in a first step, a first dielectric spacer which does not contact any of the electrically conductive layers other than the respective upper electrically conductive layer in the first step, a second contact via structure which contacts a top surface of a respective lower electrically conductive layer in the first step, and a second dielectric spacer which extends through the respective upper electrically conductive layer, and which contacts the respective lower electrically conductive layer.