Contacts to peripheral devices extending through multiple tier structures of a three-dimensional memory device can be formed with minimal additional processing steps. First peripheral via cavities through a first tier structure can be formed concurrently with formation of first memory openings. Sacrificial via fill structures can be formed in the first peripheral via cavities concurrently with formation of sacrificial memory opening fill structures that are formed in the first memory openings. Second peripheral via cavities through a second tier structure can be formed concurrently with formation of word line contact via cavities that extend to top surfaces of electrically conductive layers in the first and second tier structures. After removal of the sacrificial via fill structures, the first and second peripheral via cavities can be filled with a conductive material to form peripheral contact via structures concurrently with formation of word line contact via structures.