There are provided a method of forming a fine pattern of a semiconductor device using a silicon germanium sacrificial layer, and a method of forming a self-aligned contact using the same. The method of forming a self-aligned contact of a semiconductor device includes forming a conductive line structure having a conductive material layer, a hard mask layer, and a sidewall spacer on a substrate, and forming a silicon germanium (Si1-xGex) sacrificial layer, which has a height equal to or higher than a height of at least the conductive line structure, on an entire surface of the substrate. Then, a photoresist pattern for defining a contact hole is formed on the sacrificial layer, and the sacrificial layer is dry-etched, thereby forming a contact hole for exposing the substrate. A plurality of contacts for filling the contact hole are formed using polysilicon, and the remained sacrificial layer is wet-etched. Then, the region where the sacrificial layer is removed is filled with silicon oxide, thereby forming a first interlayer insulating layer.