A three-dimensional memory device includes a source contact layer overlying a substrate, an alternating stack of insulating layers and electrically conductive layers located overlying the source contact layer, and a memory opening fill structure located within a memory opening extending through the alternating stack and the source contact layer. The memory opening fill structure includes a composite semiconductor channel and a memory film laterally surrounding the composite semiconductor channel. The composite semiconductor channel includes a pedestal channel portion having controlled distribution of n-type dopants that diffuse from the source contact layer with a lower diffusion rate provided by carbon doping and smaller grain sizes, or has arsenic doping providing limited diffusion into the vertical semiconductor channel. The vertical semiconductor channel has large grain sizes to provide high charge carrier mobility, and is free of or includes only a low concentration of carbon atoms and n-type dopants therein.