A nonvolatile semiconductor memory device configured by a select MOS transistor provided with a gate insulator film and a select gate electrode, as well as a memory MOS transistor provided with a capacitor insulator film comprising a lower potential barrier film, a charge trapping film, and an upper potential barrier film, as well as a memory gate electrode. The charge trapping film is formed with a silicon oxynitride film and the upper potential barrier film is omitted or its thickness is limited to 1 nm and under to prevent the Gm degradation to be caused by the silicon oxynitride film, thereby lowering the erasure gate voltage. The charge trapping film is formed with a silicon oxynitride film used as a main charge trapping film and a silicon nitride film formed on or beneath the silicon oxynitride film so as to form a potential barrier effective only for holes. And, a hot-hole erasing method is employed to lower the erasure voltage.