A semiconductor memory device and a method of forming the same, the semiconductor memory device includes a substrate, a dielectric layer, plural bit lines, at least one bit line contact, a spacer structure and a spacer layer. The substrate has an isolation area to define plural active areas. The dielectric layer is disposed on the substrate, and the dielectric layer includes a bottom layer having a sidewall being retracted from sidewalls of other layers of the dielectric layer. The plural bit lines are disposed on the dielectric stacked structure, along a direction, and the at least one bit line contact is disposed below one of the bit lines, within the substrate. The spacer structure is disposed at sidewalls of each of the bit lines, and the spacer layer is disposed on the spacer structure to directly in contact with the spacer structure and the other layers of the dielectric layer.