A semiconductor device comprises a fin structure disposed over a substrate; a gate structure disposed over part of the fin structure; a source/drain structure, which includes part of the fin structure not covered by the gate structure; an interlayer dielectric layer formed over the fin structure, the gate structure, and the source/drain structure; a contact hole formed in the interlayer dielectric layer; and a contact material disposed in the contact hole. The fin structure extends in a first direction and includes an upper layer, wherein a part of the upper layer is exposed from an isolation insulating layer. The gate structure extends in a second direction perpendicular to the first direction. The contact material includes a silicon phosphide layer and a metal layer.