The performances of a semiconductor device are improved. A plurality of first gate patterns are formed over a fin of a part of a semiconductor substrate. A gate insulation film including a metal oxide film is formed between the adjacent first gate patterns. Then, a memory gate electrode is formed over the gate insulation film to fill between the adjacent first gate patterns. Then, the first gate patterns are selectively removed, to form a second gate pattern at the side surface of the memory gate electrode via the gate insulation film. Then, ions are implanted into the fin exposed from the memory gate electrode and the second gate pattern, to form an extension region in the fin. During formation of the extension region, the gate insulation film is not formed at the side surface of the fin, and hence ion implantation is not inhibited.