A method for fabricating semiconductor device includes the steps of: forming a gate structure on a substrate; forming a first recess adjacent to two sides of the gate structure; forming an epitaxial layer in the first recess; removing part of the epitaxial layer to forma second recess; and forming an interlayer dielectric (ILD) layer on the gate structure and into the second recess.