A dielectric disposable gate structure can be formed across a semiconductor material portion, and active semiconductor regions are formed within the semiconductor material portion. Raised active semiconductor regions are grown over the active semiconductor regions while the dielectric disposable gate structure limits the extent of the raised active semiconductor regions. A planarization dielectric layer is formed over the raised active semiconductor regions. In one embodiment, the dielectric disposable gate structure is removed, and a dielectric gate spacer can be formed by conversion of surface portions of the raised active semiconductor regions around a gate cavity. Alternately, an etch mask layer overlying peripheral portions of the disposable gate structure can be formed, and a gate cavity and a dielectric spacer can be formed by anisotropically etching an unmasked portion of the dielectric disposable gate structure. A replacement gate structure can be formed in the gate cavity.