In a method of manufacturing a semiconductor device, an isolation structure is formed in a substrate defining an active region, a first gate structure is formed over the isolation structure and a second gate structure over the active region adjacent to the first gate structure, a cover layer is formed to cover the first gate structure and a part of the active region between the first gate structure and the second gate structure, the active region between the first gate structure and the second gate structure not covered by the cover layer is etched to form a recess, and an epitaxial semiconductor layer is formed in the recess.