An alternating stack of insulating layers and sacrificial material layers is formed over a semiconductor material layer. Memory stack structures are formed through the alternating stack. A backside trench is formed through the alternating stack. The sacrificial material layers are replaced with electrically conductive layers. An insulating spacer is formed on sidewalls of the backside trench. A first doped semiconductor material is deposited within the backside trench. Vertical cavities are formed by vertically recessing the first doped semiconductor material at discrete locations that are laterally spaced apart. A second doped semiconductor material is deposited in the vertical cavities. The second doped semiconductor material disrupts a laterally-extending cavity in the first doped semiconductor material, thereby providing a structurally reinforced network of the first and second doped semiconductor materials for a backside contact via structure that is formed in the backside trench.