A method for fabricating semiconductor device is disclosed. The method includes the steps of: providing a substrate; forming a material layer on the substrate; forming a patterned first hard mask on the material layer; forming a patterned second hard mask on the material; utilizing the patterned first hard mask and the patterned second hard mask to remove part of the material layer for forming sacrificial mandrels; forming sidewall spacers adjacent to the sacrificial mandrels; removing the sacrificial mandrels; and using the sidewall spacers to remove part of the substrate.