A method of manufacturing a semiconductor device includes removing a part of a semiconductor substrate to form a protruding portion and a recess portion in a surface area of the semiconductor substrate, forming a first epitaxial semiconductor layer in the recess portion, forming a second epitaxial semiconductor layer on the protruding portion and the first epitaxial semiconductor layer, removing a first part of the second epitaxial semiconductor layer with a second part of the second epitaxial semiconductor layer left to expose a part of the first epitaxial semiconductor layer, and etching the first epitaxial semiconductor layer from the exposed part of the first epitaxial semiconductor layer to form a cavity under the second part of the second epitaxial semiconductor layer.