Structures and formation methods of a semiconductor device structure are provided. The semiconductor device structure includes a fin structure over a semiconductor substrate. The semiconductor device structure also includes a gate stack covering a portion of the fin structure, and the gate stack includes a work function layer and a metal filling over the work function layer. The semiconductor device structure further includes an isolation element over the semiconductor substrate and adjacent to the gate stack. The isolation element is in direct contact with the work function layer and the metal filling.