A method of fabricating semiconductor device, which reduces amount of oxidization on semiconductor substrate to suppress volume expansion of an active region of a semiconductor substrate, thereby removing pits on the semiconductor substrate. A conductive layer for forming a gate electrode and a first insulating layer serving as a mask are sequentially formed on the semiconductor substrate. Using a mask for forming a gate electrode, the first insulating layer and the conductive layer are sequentially etched to form a gate electrode. A second insulating layer and a third insulating layer are formed on the structure of the gate electrode and the surface of the semiconductor substrate. A third insulating layer formed on an overall surface of the semiconductor substrate is dry etched to form an insulating layer spacer on sidewalls of the gate electrode. A fourth insulating layer is formed on the structure of the semiconductor substrate and the gate electrode by a deposition process. That is, after forming an insulating layer spacer on sidewalls of the gate electrode, an oxide layer is formed by a deposition process so as to compensate for damage.