The present invention discloses a semiconductor device and a manufacturing method thereof. The method comprises the steps of providing a substrate on which a graphene layer or carbon nanotube layer is formed; exposing part of the graphene layer or carbon nanotube layer after forming a gate structure on the graphene layer or carbon nanotube layer, wherein the gate structure comprises a gate stack, a spacer and a cap layer, the cap layer is located on the gate stack, and the spacer surrounds the gate stack and the cap layer; epitaxially growing a semiconductor layer on the exposed graphene layer or carbon nanotube layer; and forming a metal contact layer on the semiconductor layer. In the present invention, the semiconductor layer is formed on the graphene layer or carbon nanotube layer, and then the metal contact layer is formed on the semiconductor layer, instead of forming the metal contact layer directly from the graphene layer or carbon nanotube layer. This facilitates to form the self-aligned source and drain contact plugs.