A semiconductor memory device includes an insulating film formed on a semiconductor substrate, a plurality of active areas formed on the insulating film from a semiconductor layer which is formed integrally with the substrate through openings of the insulating film, the active areas being formed by being divided into a striped shape by a plurality of trenches reaching an upper surface of the insulating film, the active areas having upper surfaces and sides respectively, a first gate insulating film formed so as to cover the upper surfaces and sides of the active areas, a charge trap layer having a face located on the first gate insulating film and confronting the upper surfaces and the sides of the active areas with the first gate insulating film being interposed therebetween, a second gate insulating film formed on the charge trap layer, and a gate electrode formed on the second gate insulating film.