The removable plug features a solid material that is housed in a porous container that has its shape changed to transition from the run in shape to the set shape. A running string and setting tool that creates relative movement deliver the plug and pull on its lower end while holding the top stationary against a backing plate. The container is pulled into itself as the radial dimension grows for the set. There can be a mandrel that remains in position and can lock to the backing plate or alternatively there can be no mandrel or a removable mandrel. The porous container can be removed in a variety of ways to let the solid material escape to be removed with fluid circulating in the wellbore. Alternatively the mandrel can be undermined to let the solid material escape for recovery.