An energy recovery system for a machine is disclosed. The energy recovery system may have a boom circuit with at least a one linear actuator configured to move a work tool, and a boom accumulator configured to selectively collect pressurized fluid from the at least one linear actuator and to discharge pressurized fluid back to the at least one linear actuator. The energy recovery system may also have a swing circuit with a swing motor configured to move the work tool, and a swing accumulator configured to selectively collect pressurized fluid from the swing motor and discharge pressurized fluid back to the swing motor. The energy recovery system may further have a common accumulator passage fluidly connecting the boom accumulator and the swing accumulator.