A semiconductor memory device comprises a MOSFET formed in the surface of a semiconductor substrate; and a trench capacitor provided in a trench formed in the surface of the semiconductor substrate. The MOSFET includes a gate electrode formed on a gate insulator in the surface of the semiconductor substrate, a sidewall insulator formed on a sidewall of the gate electrode, a first and a second diffusion layer formed on the surface of the semiconductor substrate sandwiching the gate electrode, a trench capacitor contact formed to connect the first diffusion layer to the trench capacitor, and a bit line contact formed to connect the second diffusion layer to a bit line. The sidewall insulator close to the first diffusion layer is formed thicker than the sidewall insulator close to the second diffusion layer.