A semiconductor device includes a single crystal silicon substrate an insulating layer partially formed on the single crystal silicon substrate, a single crystal silicon layer formed on the single crystal silicon substrate and the insulating layer, and containing a defect layer resulting from an excessive group IV element, and a plurality of first gate structures for memory cells, each including a first gate insulating film formed on the single crystal silicon layer, a charge storage layer formed on the first gate insulating film, a second gate insulating film formed on the charge storage layer, and a control gate electrode formed on the second gate insulating film.