In a method of manufacturing a semiconductor device, a dummy gate structure is formed over a channel region of a semiconductor layer, a source/drain epitaxial layer is formed on opposing sides of the dummy gate structure, a planarization operation is performed on the source/drain epitaxial layer, the planarized source/drain epitaxial layer is patterned, the dummy gate structure is removed to form a gate space, and a metal gate structure is formed in the gate space.