Gate electrodes of an N-channel transistor and a P-channel transistor are formed on a semiconductor substrate with a gate insulator therebetween. After conducting a first thermal treatment to the gate electrodes, N-type heavily doped diffusion layers to be a source or a drain of the N-channel transistor are formed using the gate electrode of the N-channel transistor as a mask. After conducting a second thermal treatment to the N-type heavily doped diffusion layers at a lower temperature than that of the first thermal treatment. P-type heavily doped diffusion layers to be a source or a drain of the P-channel transistor are formed using the gate electrode of the P-channel transistor as a mask. Then, a third thermal treatment is conducted to the P-type heavily doped diffusion layers at a lower temperature than that of the second thermal treatment.