A semiconductor structure includes a substrate, first channel layers vertically stacked over the substrate in a first region, and second channel layers vertically stacked over the substrate in a second region. The first and second regions have opposite conductivity types. The semiconductor structure also includes a threshold voltage (Vt) modulation layer wrapping around each of the second channel layers in the second region. The first region is free of the Vt modulation layer. The semiconductor structure also includes a gate dielectric layer wrapping around each of the first channel layers and the second channel layers over the Vt modulation layer, and a work function metal layer disposed on the gate dielectric layer and wrapping around each of the first channel layers and the second channel layers.