A semiconductor memory device comprises a plurality of columnar portions formed in memory cell array regions on a semiconductor substrate. The columnar portions are isolated from one another by a plurality of trenches, and these trenches have first and second bottoms that are different in depth. The semiconductor device comprises a plurality of cell transistors which include first diffusion layer regions formed in the first bottoms, which are shallower than the second bottoms, second diffusion layer regions formed in surface portions of the columnar portions, and a plurality of gate electrodes which are adjacent to both the first and second diffusion layer regions and extend along at least one side-surface portions of the columnar portions.