Structures and formation methods of a semiconductor device structure are provided. The method includes forming first and second fin structures over a semiconductor substrate. Each of the first and second fin structures has an upper portion and a lower portion. The method also includes forming a phosphosilicate glass (PSG) layer surrounding the upper and lower portions of the first fin structure. The method further includes doping the PSG layer to form a doped PSG layer. In addition, the method includes forming a borosilicate glass (BSG) layer surrounding the upper and lower portions of the second fin structure. The BSG layer extends over the doped PSG layer. The method also includes forming an isolation layer over the BSG layer. The method further includes partially etching the isolation layer, the BSG layer and the doped PSG layer to expose the upper portions of the first and second fin structures.