According to one embodiment, a semiconductor memory device with memory cells each composed of a vertical transistor, comprises a silicon layer formed into a columnar shape on a silicon substrate, a gate insulating film part in which a tunnel insulating film, a charge storage layer, and a block insulating film are formed to surround the sidewall surface of the silicon layer, and a stacked structure part formed to surround the sidewall surface of the gate insulating film part and in which a plurality of interlayer insulating films and a plurality of control gate electrode layers are stacked alternately. The silicon layer, gate insulating film part, and control gate electrode layer constitute the vertical transistor. The charge storage layer has a region lower in trap level than a region facing the control gate electrode layer between the vertical transistors.