The invention discloses a method for fusing a local shape feature structure and a global shape feature structure of a video image. The method comprises the following steps: describing a local shape feature by adopting a shape contour context, describing a global shape feature by adopting a contour point distribution histogram, measuring a local shape feature structure and a global shape feature structure by adopting the x<2> measure, fusing and mapping the local shape feature structure and the global shape feature structure by adopting the matrix spectrum optimization and solution method, and obtaining the feature representation. The method fills the blank of the prior art and can obtain the feature representation having strong classifying and distinguishing capacity and no unsupervised clustering.