A method of manufacturing a semiconductor device is provided. The method includes the following steps. A substrate including a first transistor having a first conductivity type, a second transistor having a second conductivity type and a third transistor having the first conductivity type is formed. An inner-layer dielectric layer is formed on the substrate, and includes a first gate trench corresponding to the first transistor, a second gate trench corresponding to the second transistor and a third gate trench corresponding to the third transistor. A work function metal layer is formed on the inner-layer dielectric layer. An anti-reflective layer is coated on the work function metal layer. The anti-reflective layer on the second transistor and on the top portion of the third gate trench is removed to expose the work function metal layer. The exposed work function metal layer is removed.