One method disclosed herein includes, among other things, forming a gate contact opening in a layer of insulating material, wherein the gate contact opening is positioned at least partially vertically above a active region, the gate contact opening exposing a portion of at least a gate cap layer of a gate structure, performing at least one etching process to remove the gate cap layer and recess a sidewall spacer so as to thereby define a spacer cavity and expose at least an upper surface of a gate electrode within the gate contact opening, filling the spacer cavity with an insulating material while leaving the upper surface of the gate electrode exposed, and forming a conductive gate contact in the gate contact opening.