A semiconductor device includes a fin structure, an isolation structure, a gate structure and an epitaxial structure. The fin structure protrudes from the surface of the substrate and includes a top surface and two sidewalls. The isolation structure surrounds the fin structure. The gate structure overlays the top surface and the two sidewalls of a portion of the fin structure, and covers a portion of the isolation structure. The isolation structure under the gate structure has a first top surface and the isolation structure at two sides of the gate structure has a second top surface, wherein the first top surface is higher than the second top surface. The epitaxial layer is disposed at one side of the gate structure and is in direct contact with the fin structure.