A method of forming a semiconductor device includes forming a first dielectric layer over a first channel region in a first region and over a second channel region in a second region; introducing a first dipole element into the first dielectric layer in the first region to form a first dipole-containing gate dielectric layer in the first region; forming a second dielectric layer over the first dipole-containing gate dielectric layer; introducing fluorine into the second dielectric layer to form a first fluorine-containing gate dielectric layer over the first dipole-containing gate dielectric layer; and forming a gate electrode over the first fluorine-containing gate dielectric layer.