An operating structure of an exhaust heat recovery device may include a bypass valve rotatable about a rotation shaft to open and close a bypass passage; a heat exchanger communicating with the bypass passage, a valve actuator inserted into the heat exchanger, and having a piston moved upward and downward depending on a temperature of coolant; a guide unit in which an end of the piston is slidably inserted, and at which an end of a rod is slidably inserted; a displacement transmission medium accommodated in the guide unit, and moved along with the piston; a link unit rotatably connected to the bypass valve and the rod, and converting sliding motion of the rod into rotation of the bypass valve; and an elastic restoring unit operating the rod so that the rod is inserted into the guide unit, or operating the bypass valve so that the bypass valve is closed.