A method of forming semiconductor memory device includes the following steps. Firstly, a substrate is provided and the substrate includes a cell region. Then, plural bit lines are disposed within the cell region along a first direction, with each of the bit line includes a tri-layered spacer structure disposed at two sides thereof. Next, plural of first plugs are formed within the cell region, with the first plugs being disposed at two sides of each bit lines. Furthermore, plural conductive patterns are formed in alignment with each first plugs. Following theses, a chemical reaction process is performed to modify the material of a middle layer of the tri-layered spacer structure, and a heat treatment process is performed then to remove the modified middle layer, thereto form an air gap layer within the tri-layered spacer structure.