A semiconductor memory device according to an embodiment comprises: a semiconductor substrate; a stacked body having a plurality of first insulating layers and conductive layers stacked alternately on the semiconductor substrate; a columnar semiconductor layer contacting the semiconductor substrate in the stacked body being provided extending in a stacking direction of the stacked body and including a first portion and a second portion which is provided above the first portion; a memory layer provided on a side surface of the columnar semiconductor layer facing the stacked conductive layers and extending along the columnar semiconductor layer; and a second insulating layer provided between one of the first insulating layer and the conductive layers of the stacked body. The columnar semiconductor layer has a boundary of the first portion and the second portion, the boundary being close to the second insulating layer; and an average value of an outer diameter of the memory layer facing a side surface of the second insulating layer is larger than that of the memory layer facing a side surface of a lowermost layer of the first insulating layers in the second portion.