A three-dimensional memory device includes an alternating stack of insulating layers and electrically conductive layers, and a memory stack structure vertically extending through the alternating stack. The memory stack structure includes a vertical semiconductor channel and a memory film. The vertical semiconductor channel can include a III-V compound semiconductor channel material. A III-V compound substrate semiconductor layer or a III-V compound semiconductor source region can be used to provide low-resistance electrical connection to a bottom end of the vertical semiconductor channel, and a drain region including a graded III-V compound semiconductor material can be used to provide low-resistance electrical connection to a top end of the vertical semiconductor channel.