A method for fabricating a semiconductor memory device is disclosed. A substrate having a main surface is provided. A memory gate is formed on the main surface of the substrate. The memory has a first sidewall and a second sidewall opposite to the first sidewall. A control gate is formed in proximity to the memory gate. The control gate has a third sidewall directly facing the second sidewall, and a fourth sidewall opposite to the third sidewall. A gap is formed between the second sidewall of the memory gate and the third sidewall of the control gate. A first single spacer structure is formed on the first sidewall of the memory gate and a second single spacer structure on the fourth sidewall of the control gate. A gap-filling layer is formed to fill up the gap.