A packer or bridge plug uses a sealing element made from a shape memory polymer (SMP). The packer element receives heat to soften the SMP while the element is compressed and retained. While so retained, the heat is removed to allow the SMP to get stiff so that it effectively seals a surrounding tubular. High expansion rates are possible as the softness of the material under thermal input allows it to be reshaped to the surrounding tubular from a smaller size during run in and to effectively retain a sealed configuration after getting stiff on reduction in its core temperature while longitudinally compressed.