A semiconductor memory device includes: a semiconductor substrate, on which an impurity diffusion layer is formed in a cell array area; a gate wiring stack body formed on the cell array area, in which multiple gate wirings are stacked and separated from each other with insulating films; a gate insulating film formed on the side surface of the gate wiring stack body, in which an insulating charge storage layer is contained; pillar-shaped semiconductor layers arranged along the gate wiring stack body, one side surfaces of which are opposed to the gate wiring stack body via the gate insulating film, each pillar-shaped semiconductor layer having the same conductivity type as the impurity diffusion layer; and data lines formed to be in contact with the upper surfaces of the pillar-shaped semiconductor layers and intersect the gate wirings.