A semiconductor device structure is provided. The semiconductor device structure includes a substrate having a base and a fin structure over the base. The fin structure has sidewalls. The semiconductor device structure includes a passivation layer over the sidewalls. The passivation layer includes dopants. The dopants include at least one element selected from group 4A elements, and the dopants and the substrate are made of different materials. The semiconductor device structure includes an isolation layer over the base and surrounding the fin structure and the passivation layer. A first upper portion of the fin structure and a second upper portion of the passivation layer protrude from the isolation layer. The semiconductor device structure includes a gate electrode over the first upper portion of the fin structure and the second upper portion of the passivation layer.