The invention discloses a multi-source heterogeneous data fusion system based on a multi-modal deep transfer learning mechanism. The system comprises a signal collector, a processor and a result output module, wherein the signal collector is used for acquiring multi-source heterogeneous information of a detection object; the processor comprises a deep feature learning module used for separately extracting source information and target information, a parameter transfer module and a pre-trained classifier or regression device, the classifier detects an object type or the regression device detects an object value; and the result output module is used for outputting a judgment result of the classifier or the regression device. According to the multi-source heterogeneous data fusion system based on the multi-modal deep transfer learning mechanism provided by the invention, the detection process adopts the classifier or the regression device, the multi-source heterogeneous data can be fused,the problem of less samples is effectively solved, the multi-modal features of the target high-rise are automatically extracted, the precision is high, and the usage is convenient.