Provided is a semiconductor memory device including a substrate, an isolation structure, a first gate dielectric layer, a first conductive layer, a second gate dielectric layer, a second conductive layer, and a protective layer. The substrate has an array region and a periphery region. The isolation structure is disposed in the substrate between the array and periphery regions. The first gate dielectric layer is disposed on the substrate in the array region. The first conductive layer is disposed on the first gate dielectric layer. The second gate dielectric layer is disposed on the substrate in the periphery region. The second conductive layer is disposed on the second dielectric layer. The second conductive layer extends to cover a portion of a top surface of the isolation structure. The protective layer is disposed between the second conductive layer and the isolation structure.