A memory device includes an alternating stack of insulating layers, dielectric barrier liners and electrically conductive layers located over a substrate and a memory stack structure extending through each layer in the alternating stack. Each of the dielectric barrier liners is located between vertically neighboring pairs of an insulating layer and an electrically conductive layer within the alternating stack. The memory stack structure includes a memory film and a vertical semiconductor channel, the memory film includes a tunneling dielectric layer and a vertical stack of discrete memory-level structures that are vertically spaced from each other without direct contact between them, and each of the discrete memory-level structures includes a lateral stack including, from one side to another, a charge storage material portion, a silicon oxide blocking dielectric portion, and a dielectric metal oxide blocking dielectric portion.