A method for manufacturing a semiconductor device includes forming a fin structure over a substrate. An isolation insulating layer is formed so that an upper part of the fin structure protrudes from the isolation insulating layer. A gate structure is formed over a part of the fin structure. Recesses are formed in the isolation insulating layer at both sides of the fin structure. A recess is formed in a portion of the fin structure which is not covered by the gate structure. The recess in the fin structure and the recesses in the isolation insulating layer are formed such that a depth D1 of the recess in the fin structure and a depth D2 of the recesses in the isolation insulating layer measured from an uppermost surface of the isolation insulating layer satisfy 0≦D1≦D2 (but D1 and D2 are not zero at the same time).