After formation of a silicon Fin part on a silicon substrate, a thin film including an impurity atom which becomes a donor or an acceptor is formed so that a thickness of the thin film formed on the surface of an upper flat portion of the silicon Fin part becomes large relative to a thickness of the thin film formed to the surface of side wall portions of the silicon Fin part. A first diagonal ion implantation from a diagonal upper direction to the thin film is performed and subsequently a second diagonal ion implantation is performed from an opposite diagonal upper direction to the thin film. Recoiling of the impurity atom from the inside of the thin film to the inside of the side wall portions and to the inside of the upper flat portion is realized by performing the first and second diagonal ion implantations.