A three-dimensional memory device includes an alternating stack of insulating layers and electrically conductive layers containing word lines and drain select gate electrodes located over a substrate, and memory stack structures containing a respective vertical semiconductor channel and a memory film including a tunneling dielectric and a charge storage layer. A first portion of a first charge storage layer located in a first memory stack structure at level of a first drain select gate electrode is thicker than a first portion of a second charge storage layer located in a second memory stack structure at the level of the first drain select electrode.