A semiconductor memory device includes an alternating stack of insulating layers and electrically conductive layers, a memory opening vertically extending through the alternating stack, and a memory opening fill structure located in the memory opening and including a vertical stack of discrete ferroelectric material portions and a vertical semiconductor channel. In one embodiment, the discrete ferroelectric material portions include a ferroelectric alloy material of a first dielectric metal oxide material and a second dielectric metal oxide material. In another embodiment, each of the discrete ferroelectric material portions is oxygen-deficient.