A method for fabricating a semiconductor device includes the steps of first forming a fin-shaped structure on a substrate, forming a dielectric layer surrounding the fin-shaped structure, performing an anneal process to transform the dielectric layer into a shallow trench isolation (STI), removing the fin-shaped structure to form a trench, and forming a stack structure in the trench. Preferably, the stack structure includes a first semiconductor layer on the fin-shaped structure and a second semiconductor layer on the first semiconductor layer and the first semiconductor layer and the second semiconductor layer include different materials.