A method of fabricating a semiconductor device includes the following steps. First, a semiconductor substrate is provided, which includes at least a fin structure and at least a gate semiconductor layer disposed thereon. The gate semiconductor layer covers a portion of the fin structure. Then a sacrificial layer is deposited to cover the fin structure entirely. Subsequently, a top surface of the fin structure is exposed from the sacrificial layer through an etching process. A material layer is then deposited, which covers the gate semiconductor layer, the fin structure and the sacrificial layer conformally. Finally, the material layer is etched until the top surface of the fin structure is exposed and a first spacer is concurrently formed on side surfaces of the gate semiconductor layer.