A method includes forming a work function metal layer over first and second semiconductor fins extending upward from a substrate; forming a sacrificial layer straddling the first semiconductor fin but not overlapping the second semiconductor fin; patterning the first work function metal layer using the sacrificial layer, resulting in a patterned work function metal layer under the sacrificial layer, and a work function metal residue in the vicinity of the second semiconductor fin; selectively forming a protective layer on a side surface of the sacrificial layer and a side surface of the patterned first work function metal layer; removing the work function metal residue after selectively forming the protective layer; after removing the work function metal residue, removing the sacrificial layer and the protective layer; and forming a second work function metal layer over the first and second semiconductor fins.