A three-dimensional semiconductor memory device includes a peripheral circuit structure on a substrate, a horizontal active layer on the peripheral circuit structure, stacks provided on the horizontal active layer to include a plurality of electrodes, a vertical structure vertically penetrating the stacks, a common source region between ones of the stacks and in the horizontal active layer, and pick-up regions in the horizontal active layer. The horizontal active layer includes first, second, and third active semiconductor layers sequentially stacked on the peripheral circuit structure. The first and third active semiconductor layers are doped to have high and low impurity concentrations, respectively, and the second active semiconductor layer includes an impurity diffusion restraining material.