A semiconductor memory device includes an alternating stack of insulating layers and electrically conductive layers, a memory opening vertically extending through the alternating stack, and a memory opening fill structure located in the memory opening and including a vertical semiconductor channel and a vertical stack of discrete ferroelectric material portions located at levels of the electrically conductive layers. The discrete ferroelectric material portions protrude inward into the memory opening relative to vertical sidewalls of the insulating layers.