The present disclosure provides a semiconductor memory device with improved element performance and reliability. The semiconductor memory device comprises a substrate, a gate electrode extending in a first direction in the substrate, a plurality of buried contacts on the substrate, and a fence in a trench between adjacent ones of the buried contacts. The fence is on the gate electrode. The fence includes a spacer film on side walls of the trench and extending in a second direction intersecting the first direction, and a filling film in the trench and on the spacer film. An upper surface of the spacer film is lower than an upper surface of the filling film with respect to the substrate.