A semiconductor memory device includes a semiconductor substrate, an isolation insulation film filled in a plurality of trenches formed in the semiconductor substrate to define a plurality of element formation regions, a floating gate of polysilicon provided on each of the element formation regions through a first insulation film, a second insulation film, provided on the floating gate, containing a metal element, a control gate of polysilicon, provided on the second insulation film, and source/drain regions provided in the semiconductor substrate, both a polysilicon conductive layer containing a metal element and a mutual diffusion layer composed of a silicate layer of a mixed oxide material composed of a silicon element contained in the floating gate and the control gate and a metal element contained in the second insulation film are provided on a surface of each of the floating gate and the control gate, respectively.