A method of forming a semiconductor memory device includes following steps. First of all, a target layer is provided, and a mask structure is formed on the target layer, with the mask structure including a first mask layer a sacrificial layer and a second mask layer. The first mask layer and the second mask layer include the same material but in different containing ratio. Next, the second mask layer and the sacrificial layer are patterned, to form a plurality of mandrels. Then, a plurality of spacer patterns are formed to surround the mandrels, and then transferred into the first mask layer to form a plurality of opening not penetrating the first mask layer. Finally, the first mask layer is used as a mask to etch the target layer, to form a plurality of target patterns.