A contact via structure vertically extending through an alternating stack of insulating layers and electrically conductive layers is provided in a staircase region having stepped surfaces. The contact via structure is electrically isolated from each electrically conductive layer of the alternating stack except for an electrically conductive layer that directly underlies a horizontal interface of the stepped surfaces. A laterally-protruding portion of the contact via structure contacts an annular top surface of the electrically conductive layer. The electrical isolation can be provided by a ribbed insulating spacer that includes laterally-protruding annular rib regions at levels of the insulating layers, or can be provided by annular insulating spacers located at levels of the electrically conductive layers. The contact via structure can contact a top surface of an underlying metal interconnect structure that overlies a substrate to provide an electrically conductive path.