An object to reduce a relief amount at the start of turning. A hydraulic drive system of a construction machine includes: a turning control valve disposed on a first circulation line extending from a first pump; a boom control valve disposed on a second circulation line extending from a second pump; first and second regulators, which change tilting angles of the first and second pumps; and a controller, which controls one or more solenoid proportional valves, which output a secondary pressure to the first and second regulators. While a turning operation is being performed, if a discharge pressure of the first pump is higher than a first setting value and a discharge pressure of the second pump is lower than a second setting value, the controller lowers first and second horsepower control lines that restrict discharge flow rates of the first and second pumps.