A three-dimensional memory device includes alternating stacks of insulating strips and electrically conductive strips located over a substrate and laterally spaced apart among one another by line trenches. The line trenches laterally extend along a first horizontal direction and are spaced apart along a second horizontal direction. Each line trench fill structure includes a laterally undulating dielectric rail having a laterally undulating width along the second horizontal direction and extending along the first horizontal direction and a row of memory stack structures located at neck regions of the laterally undulating dielectric rail. Each memory stack structure includes a vertical semiconductor channel, a blocking dielectric contacting an outer sidewall of the vertical semiconductor channel, and a charge storage layer contacting an outer sidewall of the blocking dielectric, vertically extending continuously through each level of the electrically conductive strips, and having a vertically undulating lateral thickness.