A method of manufacturing a Fin FET includes forming a fin structure including an upper layer. Part of the upper layer is exposed from an isolation insulating layer. A dummy gate structure is formed over part of the fin structure. The dummy gate structure includes a dummy gate electrode layer and a dummy gate dielectric layer. An interlayer insulating layer is formed over the dummy gate structure. The dummy gate structure is removed so that a space is formed. A gate dielectric layer is formed in the space. A first metal layer is formed over the gate dielectric in the space. A second metal layer is formed over the first metal layer in the space. The first and second metal layers are partially removed, thereby reducing a height of the first and second metal layers. A third metal layer is formed over the partially removed first and second metal layers.