An isolation structure of a semiconductor device is formed by forming a hard mask layer on a semiconductor substrate having active and field regions to expose the field region. A trench is defined by etching the exposed field region of the semiconductor substrate using the hard mask as an etch mask. An SOG layer is formed in the trench partially filling the trench. An amorphous aluminum oxide layer is formed on the resultant substrate including the SOG layer. An HDP layer is formed on the amorphous aluminum oxide layer to completely fill the trench. The HDP layer and the amorphous aluminum oxide layer are subjected to CMP to expose the hard mask. The hard mask and portions of the amorphous aluminum oxide layer that are formed on the HDP layer are removed. The amorphous aluminum oxide layer is crystallized.