A method of manufacturing a semiconductor device includes forming a stacked structure of first semiconductor layers and second semiconductor layers alternately stacked in a first direction over a substrate. A thickness of the first semiconductor layers as formed increases in each first semiconductor layer spaced further apart from the substrate in the first direction. The stacked structure is patterned into a fin structure extending along a second direction substantially perpendicular to the first direction. A portion of the first semiconductor layers between adjacent second semiconductor layers is removed, and a gate structure is formed extending in a third direction over a first portion of the first semiconductor layers so that the gate structure wraps around the first semiconductor layers. The third direction is substantially perpendicular to both the first direction and the second direction. Each of the first semiconductor layers at the first portion of the first semiconductor layers have a substantially same thickness.