A three-dimensional memory device includes driver transistors containing boron doped semiconductor active regions, device contact via structures in physical contact with the boron doped semiconductor active regions, the device contact via structures containing at least one of tantalum, tungsten, and cobalt, and a three-dimensional memory array located over the driver transistors and including an alternating stack of insulating layers and electrically conductive layers and memory structures vertically extending through the alternating stack.