A method of manufacturing a semiconductor device including following steps is provided. A substrate is provided. An ion implantation process is performed on the substrate to form doped material layers at different depth positions of the substrate and to define at least one nanowire layer. The at least one nanowire layer and the doped material layers are alternately stacked. A patterning process is performed on the at least one nanowire layer and the doped material layers to form at least one nanowire and doped layers. The at least one nanowire and the doped layers are alternately stacked to form a stack structure. A dummy gate structure spanning over the stack structure is formed. Spacers located on sidewalls of the dummy gate structure is formed. The dummy gate structure is removed to expose the at least one nanowire and the doped layers. The exposed doped layers are removed to form openings.