A semiconductor memory device includes a semiconductor substrate, an isolation insulation film filled in a plurality of trenches formed in the semiconductor substrate to define a plurality of element formation regions, a floating gate provided on each of the element formation regions through a first gate insulation film, a control gate provided on the floating gate through a second gate insulation film, and source/drain regions provided in the semiconductor substrate, wherein a mutual diffusion layer is provided at least at an interface between the second gate insulation film and the control gate.