A method for fabricating semiconductor device with fin-shaped structure is disclosed. The method includes the steps of: forming a fin-shaped structure on a substrate; forming a first dielectric layer on the substrate and the fin-shaped structure; depositing a second dielectric layer on the first dielectric layer; etching back a portion of the second dielectric layer; removing part of the first dielectric layer to expose a top surface and part of the sidewall of the fin-shaped structure; forming an epitaxial layer to cover the exposed top surface and part of the sidewall of the fin-shaped structure; and removing a portion of the second dielectric layer.