A three-dimensional memory device includes an alternating stack of insulating layers and electrically conductive layers located over a substrate, and a memory stack structure extending through the alternating stack. The memory stack structure includes a vertical semiconductor channel, a vertical stack of majority germanium layers each containing at least 51 atomic percent germanium, and a vertical stack of ferroelectric dielectric layers.