The present invention relates to a physical audit system with radio frequency identification (RFID) and a method thereof, which perform physical audit by means of RFID. First, a plurality of electronic tags is set in a physical system and all the physical units included in the physical system, respectively. The plurality of electronic tags records physical configuration data related to the physical system and the physical units, respectively. A radio-frequency writer unit writes the physical configuration data of a newer version according to the physical system and the physical units. Then, a radio-frequency reader unit reads the physical configuration data of the plurality of electronic tags, and transmits the physical configuration data to an integrated data processing device for analyzing and comparing the physical configuration data. It is judged if the physical configuration is identical to comparison data in the integrated data processing device.