A method of removing a dummy gate dielectric layer is provided. Firstly a first plasma containing F is utilized to remove the dummy dielectric layer which contains Si and O. Then a second plasma containing H2 is utilized to remove fluorine compound on the surface of the semiconductor substrate. Since the fluorine residue formed after the first plasma treatment reacts with the second plasma to form a gaseous product HF, the fluorine element can be taken away from the semiconductor device with the HF, which prevents inversion layer offset and gate current leakage occurred in the subsequent processing steps due to the fluorine element.