A method of forming a semiconductor memory device includes following steps. First of all, a dielectric layer is formed on a semiconductor substrate, and a conductive pad is formed in the dielectric layer. Then, a stacked structure is formed on the dielectric layer, and the stacked structure includes a first layer, a second layer and a third layer stacked one over another on the conductive pad. Next, a patterned mask layer is formed on the stacked structure, and a portion of the stacked structure is removed, to form an opening in the stacked structure, with the opening having a tapered sidewall in the second layer and the first layer. After that, the tapered sidewall of the opening in the second layer is vertically etched, to form a contact opening in the stacked structure. Finally, the patterned mask layer is removed.