One illustrative device disclosed herein includes, among other things, an active region defined in a semiconductor substrate, a layer of material positioned above the substrate, a plurality of laterally spaced-apart source/drain trenches formed in the layer of material above the active region, a conductive source/drain contact structure formed within each of the source/drain trenches, a gate trench formed at least partially in the layer of material between the spaced-apart source/drain trenches in the layer of material, wherein portions of the layer of material remain positioned between the source/drain trenches and the gate trench, a gate structure positioned within the gate trench, and a gate cap layer positioned above the gate structure.