A three-dimensional memory device includes an alternating stack of insulating layers and electrically conductive layers, memory opening fill structures extending through the alternating stack, where each of the memory opening fill structures includes a vertical semiconductor channel, a tunneling dielectric layer, and a vertical stack of memory elements located at levels of the electrically conductive layers between a respective vertically neighboring pair of the insulating layers. Each of the memory elements is located at a level of a respective one of the electrically conductive layers between the respective vertically neighboring pair of the insulating layers. Each of the memory elements includes a first memory material portion, and a second memory material portion that is vertically spaced from the first memory material portion. The second memory material portion has a different material composition from the first memory material portion.