A semiconductor device includes a substrate, a gate stack, and epitaxy structures. The substrate has a P-type region. The gate stack is over the P-type region of the substrate and includes a gate dielectric layer, a bottom work function (WF) metal layer, a top WF metal layer, and a filling metal. The bottom WF metal layer is over the gate dielectric layer. The top WF metal layer is over and in contact with the bottom WF metal layer. Dipoles are formed between the top WF metal layer and the bottom WF metal layer, and the dipoles direct from the bottom WF metal layer to the top WF metal layer. The filling metal is over the top WF metal layer. The epitaxy structures are over the P-type region of the substrate and on opposite sides of the gate stack.