Void formation in tungsten lines in a three-dimensional memory device can be prevented by providing polycrystalline aluminum oxide liners in portions of lateral recesses that are laterally spaced from backside trenches by a distance grater than a predefined lateral offset distance. Tungsten nucleates on the polycrystalline aluminum oxide liners prior to nucleating on a metallic liner layer. Thus, tungsten layers can be deposited from the center portion of each backside recess, and the growth of tungsten can proceed toward the backside trenches. By forming the tungsten layers without voids, structural integrity of the three-dimensional memory device can be enhanced.