A semiconductor memory device includes a stack of alternating insulating layers and first conductive layers disposed over a substrate; a plurality of memory cell strings penetrating the stack over the substrate, each memory cell string comprising a central portion extending through the stack, a semiconductor layer surrounding the central portion, and a ferroelectric layer surrounding the semiconductor layer, and the central portion comprising a channel isolation structure and a second conductive layer and a third conductive layer at two sides of the channel isolation structure; and a plurality of cell isolation structures penetrating the conductive layers and the insulating layers over the substrate and disposed between two memory cell strings, each cell isolation structure comprising a top portion and a bottom portion adjoined to the top portion and different from the top portion.