A semiconductor device comprising a memory cell which includes: a pillar-shaped semiconductor layer of a first conductive type formed on a semiconductor substrate; source and drain diffusion layers of a second conductive type formed in upper and lower portions of the pillar-shaped semiconductor layer; a semiconductor layer of the second conductive type or a cavity formed inside the pillar-shaped semiconductor layer; and a gate electrode formed on a side face of the pillar-shaped semiconductor layer via a gate insulating film, or a control gate electrode formed on the side face of the pillar-shaped semiconductor layer via a charge accumulation layer.