A semiconductor memory device includes a memory gate disposed on a main surface of a substrate. The memory has a first sidewall and a second sidewall opposite to the first sidewall. A control gate is in proximity to the memory gate. The control gate has a third sidewall directly facing the second sidewall, and a fourth sidewall opposite to the third sidewall. A gap is provided between the second sidewall of the memory gate and the third sidewall of the control gate. A first single spacer structure is disposed on the first sidewall of the memory gate. A second single spacer structure is disposed on the fourth sidewall of the control gate. A gap-filling layer is deposited into the gap and fills up the gap.