When a dummy sidewall and source and drain regions are once formed and then the dummy sidewall is removed to extend the source and drain regions, the removal of the dummy sidewall is performed after formation of a protective oxide film on a gate electrode and on the major surfaces of the source and drain regions. This efficiently prevents conventional surface roughness of the upper surface of the gate electrode and the impurity region due to the removal of the dummy sidewall.