A hydraulic control system is disclosed for use with a machine. The hydraulic control system may have a tank, a pump, a swing motor, and at least one control valve configured to control fluid flow between the pump, the swing motor, and the tank. The hydraulic system may also have an accumulator configured to selectively receive pressurized fluid discharged from the swing motor and selectively supply pressurized fluid to the swing motor, at least one accumulator valve, and a controller. The controller may be configured to receive input indicative of a difference between desired and actual speeds of the swing motor, and determine if the swing motor is accelerating or decelerating based on the difference. The controller may also be configured to control the at least one accumulator valve to cause the accumulator to selectively receive or supply pressurized fluid only when the swing motor is accelerating or decelerating.