A passive seat belt system comprises a seat belt adapted to reciprocate between a fasten end and a release end so as to be fastened and released to and from a rider, a motor for driving the seat belt, and a controller for controlling the motor so as to fasten and release the seat belt to and from the rider as he enters and exits the vehicle. The controller controls the motor in such a manner that the motor is stopped and then reversely rotated after the lapse of a predetermined period when the motor is to be reversely rotated during rotation thereof.