A semiconductor memory device comprises a semiconductor substrate having memory cell area, a plurality of trenches selectively formed in the memory cell area aligning in certain intervals and a plurality of memory cell arrays provided in the memory cell area, wherein each of the memory cell arrays comprises a plurality of MOS transistors connected in a serial array and a plurality of capacitors each formed in a corresponding one of the trenches. Each of the transistors has a gate electrode above the substrate with a gate insulating film formed therebetween and source and drain regions formed in the substrate on both sides of the gate electrode. Each of the capacitors includes a charge storage layer formed on an inner wall of each of the trenches and connected integrally to one of the source and drain regions of each of the transistors, a capacitor insulating film formed on the charge storage layer and a capacitor electrode formed on the capacitor insulating film so as to bury each of the trenches and extending to the surface of the substrate, which is formed on the surface of the substrate except for at least formation areas of the transistors.