A stacked semiconductor device that is reliable by forming an insulating layer on a lower memory layer and by forming a single crystalline semiconductor in portions of the insulating layer. A method of manufacturing the stacked semiconductor device, including: providing a lower memory layer including a plurality of lower memory structures; forming an insulating layer on the lower memory layer; forming trenches by removing portions of the insulating layer; forming a preparatory semiconductor layer for filling the trenches; and forming a single crystalline semiconductor layer by phase-changing the preparatory semiconductor layer.