An improvement is achieved in the performance of a semiconductor device including a memory element. Over a semiconductor substrate, a gate electrode for the memory element is formed via an insulating film as a gate insulating film for the memory element. The insulating film includes first, second, third, fourth, and fifth insulating films in order of being apart from the substrate. The second insulating film has a charge storing function. The band gap of each of the first and third insulating films is larger than a band gap of the second insulating film. The band gap of the fourth insulating film is smaller than the band gap of the third insulating film. The band gap of the fifth insulating film is smaller than the band gap of the fourth insulating film.