A method of forming a gate structure on a fin structure includes the steps of providing a fin structure covered by a first silicon oxide layer, a silicon nitride layer, a gate material and a cap material in sequence, wherein the silicon nitride layer contacts the first silicon oxide layer. Later, the cap material is patterned to form a first cap layer and the gate material is patterned to form a first gate electrode by taking the silicon nitride layer as an etching stop layer. Then, the silicon nitride layer not covered by the first gate electrode is removed to expose part of the first silicon oxide layer. Finally, a first dielectric layer is formed to conformally cover the first silicon oxide layer, the first gate electrode and the first cap layer.