The performances of a semiconductor device of a memory element are improved. Over a semiconductor substrate, a gate electrode for memory element is formed via overall insulation film of gate insulation film for memory element. The overall insulation film has first insulation film, second insulation film over first insulation film, third insulation film over second insulation film, fourth insulation film over third insulation film, and fifth insulation film over fourth insulation film. The second insulation film is an insulation film having charge accumulation function. Each band gap of first insulation film and third insulation film is larger than the band gap of second insulation film. The third insulation film is polycrystal film including high dielectric constant material containing metallic element and oxygen. Fifth insulation film is polycrystal film including the same material as that for third insulation film. Fourth insulation film includes different material from that for third insulation film.