In a method for manufacturing a semiconductor device, a gate structure is formed over a channel layer and an isolation insulating layer. A first sidewall spacer layer is formed on a side surface of the gate structure. A sacrificial layer is formed so that an upper portion of the gate structure with the first sidewall spacer layer is exposed from the sacrificial layer and a bottom portion of the gate structure with the first sidewall spacer layer is embedded in the first sacrificial layer. A space is formed between the bottom portion of the gate structure and the sacrificial layer by removing at least part of the first sidewall spacer layer. After the first sidewall spacer layer is removed, an air gap is formed between the bottom portion of the gate structure and the sacrificial layer by forming a second sidewall spacer layer over the gate structure.