A semiconductor device and a method of forming the same, the semiconductor device includes a fin shaped structure, agate structure, an epitaxial layer, an interlayer dielectric layer, a first plug and a protection layer. The fin shaped structure is disposed on a substrate, and the gate structure is across the fin shaped structure. The epitaxial layer is disposed in the fin shaped structure, adjacent to the gate structure. The interlayer dielectric layer covers the substrate and the fin shaped structure. The first plug is formed in the interlayer dielectric layer, wherein the first plug is electrically connected to the epitaxial layer. The protection layer is disposed between the first plug and the gate structure.