A system and method of detecting whether a seat belt is worn is disclosed. The system includes a contact type sensor and a non-contact type sensor installed in a seat belt buckle. A Body Control Module (BCM) receives a signal from the contact type sensor and transmitting the signal to a cluster. An airbag control unit receives a signal from the non-contact type sensor. The seat belt control unit determines whether a seat belt is worn using a signal received from the cluster when an ignition key is off, and determines whether the seat belt is worn using a signal received from the airbag control unit when the ignition key is on.