A semiconductor device and a method of manufacturing the same, the semiconductor device includes a fin shaped structure, a gate structure, an epitaxial layer, a germanium layer, an interlayer dielectric layer and a first plug. The fin shaped structure is disposed on a substrate. The gate structure is formed across the fin shaped structure. The epitaxial layer is disposed in the fin shaped structure adjacent to the gate structure. The germanium layer is disposed on the epitaxial layer. The interlayer dielectric layer covers the substrate and the fin shaped structure. The first plug is disposed in the interlayer dielectric layer to contact the germanium layer.