A semiconductor device includes a first gate structure, a second gate structure, a first source/drain structure and a second source/drain structure. The first gate structure includes a first gate electrode and a first cap insulating layer disposed on the first gate electrode. The second gate structure includes a second gate electrode and a first conductive contact layer disposed on the first gate electrode. The first source/drain structure includes a first source/drain conductive layer and a second cap insulating layer disposed over the first source/drain conductive layer. The second source/drain structure includes a second source/drain conductive layer and a second conductive contact layer disposed over the second source/drain conductive layer.