A semiconductor memory device includes a substrate, a dielectric layer on the substrate, and a contact plug in the dielectric layer. An upper portion of the contact plug protrudes from a top surface of the dielectric layer. The upper portion of the contact plug acts as a first electrode. A buffer layer is disposed on the dielectric layer and beside the upper portion of the contact plug. A resistive-switching layer is disposed beside the buffer layer. A second electrode is disposed beside the resistive-switching layer.