A semiconductor memory device includes a substrate, plural gates, plural cell plugs, a capacitor structure and a stacked structure. The gates are disposed in the substrate, and the cell plugs are disposed on the substrate, to electrically connect the substrate at two sides of each gate. The capacitor structure includes plural capacitors, and each capacitor is electrically connected each cell plug. The stacked structure covers the capacitor structure, and the stacked structure includes a semiconductor layer, a conductive layer on the semiconductor layer and an insulating layer stacked on the conductive layer. Two gaps are defined respectively between a side portion of the insulating layer and a lateral portion of the conductive layer at two sides of the capacitor structure, and the two gaps have different lengths.