After forming a gate stack straddling a portion of each semiconductor fin of a plurality of semiconductor fins located over a substrate, a gate liner is formed on sidewalls of a lower portion of the gate stack that contacts the plurality of semiconductor fins and a gate spacer having a width greater than a width of the gate liner is formed on sidewalls of an upper portion of the gate stack that is located above the plurality of semiconductor fins. The width of the gate spacer thus is not limited by the fin pitch, and can be optimized to improve the device performance.