A stacked semiconductor device has three-dimensional alternate layers of iconductor elements and insulating layers each electrically insulating the adjacent upper and lower layers of semiconductor elements, formed on a single crystal semiconductor substrate. A semiconductor is deposited in openings formed respectively in the insulating layers to form single crystal semiconductor layers each having the same crystal axis as the single crystal semiconductor substrate respectively over the insulating layers, and semiconductor elements are formed respectively in a plurality of layers. The opening formed through the upper insulating layer reaches the lower layer of the semiconductor element immediately below the same upper insulating layer, and is formed at a position spaced apart horizontally from the opening formed through the lower insulating layer immediately below the same upper insulating layer. A semiconductor for forming the upper layer of a semiconductor having the same crystal axis as the lower layer of a semiconductor is deposited in the opening of the upper insulating layer so that satisfactory lateral epitaxial growth will occur over the insulating layer.