A first gate layer of a first conductor layer is formed on a gate oxide layer and selectively covered with a second gate layer of a second conductor layer. The first and second gate layers are used as a mask and a semiconductor substrate is thermally oxidized to thereby increase a thickness of a portion of the gate oxide layer except the gate layers and cover the second gate layer and the portion having the increased thickness of the gate oxide layer with a third conductor layer. Thereafter, a side wall of an insulating layer is formed on a side portion of the second gate layer and is used as a mask to form a third gate layer. The first, second and third gate layers and the side wall are used as a mask and impurity is introduced into the semiconductor substrate relatively heavily to thereby form a heavily doped impurity layer.