A semiconductor device with a contact structure includes a silicon substrate, a diffusion region formed in a surface of the silicon substrate, a silicide film of high melting point metal deposited on the diffusion region, an insulating film formed on the silicon substrate, a contact hole formed in the insulating film such that the silicide film is exposed at a bottom of the contact hole, an anti-diffusion film formed on the exposed surface of the silicide film at the bottom of the contact film, a plug formed in the contact hole by a selective Al CVD, and a metal wiring formed on the insulating film such that the metal wiring is electrically connected to the diffusion region by means of the plug, anti-diffusion film and silicide film. The anti-diffusion film may be formed by nitriding the surface of the silicide film.