Structures and formation methods of a semiconductor device structure are provided. The semiconductor device structure includes a first semiconductor layer and a second semiconductor layer vertically stacked over a semiconductor substrate. The first semiconductor layer and the second semiconductor layer include different materials. The semiconductor device structure also includes a gate stack covering a first portion of the first semiconductor layer. The semiconductor device structure further includes a spacer element over a sidewall of the gate stack. The spacer element covers the second semiconductor layer and a second portion of the first semiconductor layer. The thickness of the second semiconductor layer is different from the thickness of the second portion.