An energy recovery system is disclosed for use with a hydraulic machine. The energy recovery system may have a tank, a pump configured to draw fluid from the tank and pressurize the fluid, an actuator, and an actuator control valve movable to direct pressurized fluid from the pump to the actuator and from the actuator to the tank to move the actuator. The energy recovery system may also have a motor mechanically connected to a rotary device and configured to selectively receive fluid discharged from the actuator, and at least one valve movable to selectively redirect fluid exiting the motor back to the actuator.