There is provided a method of fabricating a semiconductor memory device having a self-aligned contact, including the steps of forming a plurality of gate electrodes by interposing a gate insulating layer on an active region of a semiconductor substrate in a predetermined direction at constant intervals, forming a first insulating layer on the resultant structure having the gate electrodes and then forming one or more of each of first and second openings which partially open an active region of the semiconductor substrate, forming first and second pad layers by filling the first and second openings with a conductive material, forming a first interlayer dielectric film on the first insulating layer having the first and second pad layers and forming a third opening which opens the surface of the first pad layer, forming a plurality of bit lines in a direction orthogonal to the gate electrodes on the first interlayer dielectric film while filling the third opening, depositing an insulating layer on the resultant structure having the bit lines and removing the insulating layer on the bit lines and on the first interlayer dielectric film to form insulating spacers only at both side walls of the bit lines, forming a second interlayer dielectric film on the resultant structure having the insulating spacers and forming a fourth opening self-aligned to the insulating spacers to open the surface of the second pad layer, and filling the fourth opening with a conductive material.