A three-dimensional memory device includes an alternating stack of source layers and drain layers located over a substrate, a memory opening vertically extending through the alternating stack, a vertical word line located in the memory opening and vertically extending through each of the source layers and the drain layers of the alternating stack, discrete semiconductor channels contacting horizontal surfaces of a respective vertically neighboring pair of a source layer of the source layers and a drain layer of the drain layers, and a vertical stack of discrete memory material portions laterally surrounding the vertical word line.