An element isolating region for separating an element region of a semiconductor layer is formed in a peripheral circuit section of a semiconductor memory device, and a first conductive layer is formed with the element region with a first insulating film interposed therebetween. A second conductive layer is formed on the first conductive layer to extend into the element isolating region. A surface of that section of the second conductive layer which is positioned within the element isolating region is exposed, and a third conductive layer is formed on the second conductive layer with a second insulating film interposed therebetween. Further, a contact is electrically connected to an exposed surface of the second conductive layer.