A three-dimensional memory device includes alternating stacks of insulating strips and electrically conductive strips laterally spaced apart among one another by line trenches and a two-dimensional array of memory stack structures and a two-dimensional array of dielectric pillar structures located in the line trenches. Each line trench is filled with laterally alternating sequence of memory stack structures and dielectric pillar structures. Each memory stack structure contains a vertical semiconductor channel, a pair of blocking dielectrics contacting outer sidewalls of the vertical semiconductor channel, a pair of charge storage layers contacting outer sidewalls of the pair of blocking dielectrics, and a pair of tunneling dielectrics contacting outer sidewalls of the pair of charge storage layers.