A semiconductor device adopting an interlayer contact structure between upper and lower conductive layers and a method of manufacturing the semiconductor device adopting the structure are provided. The lower conductive layer includes a first conductive layer and a first silicide layer stacked together. The upper conductive layer includes a second conductive layer doped with impurities and a second silicide layer stacked together. In the interlayer contact structure, the first and second conductive layers are in direct contact with each other. This decreases the contact resistance between the two conductive layers and improves the electrical properties of the device.