The invention provides a steer-by-wire system and a stability control method thereof. The steer-by-wire system comprises a steering wheel module, a front wheel steering module and an ECU electronic control unit, wherein the ECU electronic control unit comprises a variable transmission ratio controller, a fuzzy self-adaptive PID controller and a full state feedback controller. The stability control method is as follows: the variable transmission ratio controller calculates an ideal transmission ratio at the moment according to a steering wheel rotation angle signal and a vehicle speed signal, to obtain an ideal front wheel rotation angle at the moment; the full state feedback controller of an automobile calculates a compensation rotation angle of front wheels according to a yaw rate signal and a lateral acceleration signal of the automobile, to realize the outer ring control of the steer-by-wire system; and finally the fuzzy self-adaptive PID controller is designed, to realize the inner ring control of the steer-by-wire system. The operating stability of the automobile is effectively improved through the coordinated control of inner and outer rings of the steer-by-wire system.