A semiconductor memory device has a semiconductor substrate, a plurality of word lines formed at predetermined intervals on the semiconductor substrate, each word line having a gate insulating film, a charge storage layer, a first insulating film, and a controlling gate electrode which are stacked in order, and including a metal oxide layer above the level of the gate insulating film, a second insulating film covering a side of the word line and a surface of the semiconductor substrate between the word lines, and having a film thickness of 15 nm or less, and a third insulating film formed between the word lines adjacent to each other such that a region below the level of the metal oxide layer has a cavity.