After formation of a gate cavity straddling at least one semiconductor material portion, a gate dielectric layer and at least one work function material layer is formed over the gate dielectric layer. The at least one work function material layer and the gate dielectric layer are patterned such that remaining portions of the at least one work function material layer are present only in proximity to the at least one semiconductor material portion. A conductive material having a greater conductivity than the at least one work function material layer is deposited in remaining portions of the gate cavity. The conductive material portion within a replacement gate structure has the full width of the replacement gate structure in regions from which the at least one work function material layer and the gate dielectric layer are removed.