In one embodiment, an autonomous driving vehicle (ADV) speed following system determines how much and when to apply a throttle or a brake control of an ADV to maneuver the ADV around, or to avoid, obstacles of a planned route. The speed following system calculates a first torque force to accelerate the ADV, a second torque force to counteract frictional forces and wind resistances to maintain a reference speed, and a third torque force to minimize an initial difference and external disturbances thereafter between predefined target speed and actual speed of the ADV over a planned route. The speed following system determines a throttle-brake torque force based on the first, second, and third torque forces and utilizes the throttle-brake torque force to control a subsequent speed of the ADV.