The invention discloses a transformer substation-oriented machine learning-based semantic labeling method. The method comprises the following steps of: extracting features of a training video and a test image; clustering the features; associating and quantifying generated visual words; carrying out learning by a machine model according to the video and the image after feature description; and labeling a test video and a picture by using a semi-supervised or weak-supervised learning method. According to the method, field operation standardization and data standardization can be realized, data and analysis algorithm sharing between a field side and a center side is realized, the data analysis ability of the field side is improved, the long-distance control and real-time technology support abilities of field operation and maintenance are strengthened, the professionalization and intelligence levels of the field operation and maintenance are improved, the inspection working efficiency andequipment state control ability are enhanced, and the condition based maintenance and aid decision making are strengthened.