An illustrative device disclosed herein includes at least one layer of insulating material, a conductive contact structure having a conductive line portion and a conductive via portion and a memory cell positioned in a first opening in the at least one layer of insulating material. In this illustrative example, the memory cell includes a bottom electrode, a memory state material positioned above the bottom electrode and an internal sidewall spacer positioned within the first opening and above at least a portion of the memory state material, wherein the internal sidewall spacer defines a spacer opening and wherein the conductive via portion is positioned within the spacer opening and above a portion of the memory state material.