A semiconductor device includes a first gate structure in a substrate and a second gate structure in the substrate and adjacent to the first gate structure. Preferably, a top surface of the first gate structure and a top surface of the second gate structure are lower than a top surface of the substrate and a number of work function metal layers in the first gate structure and the second gate structure are different.