A method of forming a semiconductor memory device is disclosed. A top electrode layer is formed on the MTJ stack layer. A patterned buffer layer is formed to cover only the logic circuit region. A hard mask layer is formed on the top electrode layer and the patterned buffer layer. A patterned resist layer is formed on the hard mask layer. A first etching process is performed to etch the hard mask layer and the top electrode layer not covered by the patterned resist layer in the memory region and the hard mask layer, the patterned buffer layer and the top electrode layer in the logic circuit region, thereby forming a top electrode on the MTJ stack layer in the memory region and a remaining top electrode layer covering only the logic circuit region on the MTJ stack layer.