Embodiments of mechanisms for forming a semiconductor device are provided. The semiconductor device includes a semiconductor substrate. A source region and a drain region are formed in the semiconductor substrate, and metal silicide regions are formed in the source region and the drain region, respectively. The semiconductor device further includes a metal gate stack formed over the semiconductor substrate and between the source region and the drain region. The semiconductor device also includes an insulating layer formed over the semiconductor substrate and surrounding the metal gate stack, wherein the insulating layer has contact openings exposing the metal silicide regions, respectively. The semiconductor device includes a dielectric spacer liner layer formed over inner walls of the contact openings, wherein the whole of the dielectric spacer liner layer is right above the metal silicide regions. The semiconductor device includes contact plugs formed in the contact openings.