Structures and formation methods of a semiconductor device structure are provided. The semiconductor device structure includes a fin structure over a semiconductor substrate and a gate stack covering a portion of the fin structure. The gate stack includes a work function layer and a gate dielectric layer. The semiconductor device structure also includes an isolation element over the semiconductor substrate and adjacent to the gate stack. The isolation element is in direct contact with the work function layer and the gate dielectric layer, and a lower width of the isolation element is greater than an upper width of the isolation element.