A three-dimensional memory device includes alternating stacks of insulating strips and electrically conductive strips laterally spaced apart by line trenches, and an alternating two-dimensional array of memory stack assemblies and dielectric pillar structures located in the line trenches. Each of the line trenches is filled with a respective laterally alternating sequence of memory stack assemblies and dielectric pillar structures. Each memory stack assembly includes a vertical semiconductor channel and a pair of memory film. The vertical semiconductor channel includes a semiconductor channel layer having large grains, which can be provided by a selective semiconductor growth from seed semiconductor material layers, sacrificial semiconductor material layers, or a single crystalline semiconductor material in a semiconductor substrate underlying the alternating stacks.