A pump torque control system is capable of preventing from hunting due to interference between speed sensing control and control on an engine speed of a prime mover when the temperature of a hydraulic fluid is low, and performing appropriate pump torque control. A regulator 31 controls displacement volumes of hydraulic pumps 2 and 3 to ensure that the total absorption torque of the hydraulic pumps 2 and 3 does not exceed a set maximum absorption torque. An engine sensor 33, a hydraulic fluid temperature sensor 34, a solenoid proportional valve 35, and a controller 23 perform the speed sensing control to ensure that the maximum absorption torque of the hydraulic pumps 2 and 3 is reduced. The maximum absorption torque is set in the regulator 31 based on a deviation between a target engine speed of the prime mover 1 and an actual engine speed of the prime mover 1. A second modification factor calculating section 45 and a control gain modifying section 49, which are included in the controller 23, change a control gain of the speed sensing control based on a value detected by the hydraulic temperature sensor 34 to ensure that the control gain is reduced as the temperature of the hydraulic fluid is reduced.