A hydraulic drive system of a construction machine includes: a control valve configured such that an opening area of a center bypass passage gradually decreases in accordance with increase in an operation signal; an operation device that outputs the operation signal; a bleed-off line; and a bleed-off valve that includes a pilot port to which a secondary pressure from a solenoid proportional valve is led, the bleed-off valve being configured such that: a bypass passage is open until the operation signal exceeds a first setting value, and the bypass passage is closed when the operation signal exceeds the first setting value; and an unloading passage is opened when the operation signal exceeds a second setting value, and an opening area of the unloading passage gradually decreases until the operation signal that has exceeded the second setting value reaches a third setting value.