A semiconductor structure includes a semiconductor device, a hydrogen diffusion barrier layer, a lower metal line structure located below the hydrogen diffusion barrier layer, an alternating stack of insulating layers and electrically conductive layers, memory stack structures vertically extending through the alternating stack in a memory array region, a through-stack contact via structure extending through the alternating stack and through the hydrogen diffusion barrier layer in the memory array region and contacting the lower metal line structure, and a through-stack insulating spacer laterally surrounding the through-stack contact via structure and extending through the alternating stack but not extending through the hydrogen diffusion barrier layer.