Semiconductor device and the manufacturing method thereof are disclosed. An exemplary semiconductor device comprises first semiconductor layers and second semiconductor layers over a substrate, wherein the first semiconductor layers and the second semiconductor layers are separated and stacked up, and a thickness of each second semiconductor layer is less than a thickness of each first semiconductor layer; a first interfacial layer around each first semiconductor layer; a second interfacial layer around each second semiconductor layer; a first dipole gate dielectric layer around each first semiconductor layer and over the first interfacial layer; a second dipole gate dielectric layer around each second semiconductor layer and over the second interfacial layer; a first gate electrode around each first semiconductor layer and over the first dipole gate dielectric layer; and a second gate electrode around each second semiconductor layer and over the second dipole gate dielectric layer.