A method of forming a three-dimensional memory device includes forming memory stack structures vertically extending through an alternating stack of insulating layers and electrically conductive layers over a substrate, such that each of the memory stack structures includes a memory film and a vertical semiconductor channel laterally surrounded by the memory film. The method also includes forming a stack of a first silicon nitride layer and a second silicon nitride layer over the memory stack structures, such that the first silicon nitride layer has a higher hydrogen-to-nitrogen ratio than the second silicon nitride layer, performing an anneal process at an elevated temperature to diffuse hydrogen from the first silicon nitride layer into the memory stack structures, and removing the first and second silicon nitride layers.