The invention discloses a dual-mode video emotion recognition method with a composite spatial-temporal characteristic. The method comprises the following steps: 1, extending an existing volume local binary pattern algorithm into a spatial-temporal ternary pattern to obtain spatial-temporal local ternary pattern moment texture characteristics of a human face expression and an upper body posture; 2, enhancing the description of an emotional video by further fusing characteristics of a three-dimensional gradient direction histogram in order to compensate for the lack of expression of the texture characteristics for image edges and direction information, and combining the two characteristics into the composite spatial-temporal characteristic; and (3) fusing information of the two modes according to a D-S evidence combination rule to obtain an emotion recognition result. By the use of the composite spatial-temporal characteristic for full description of an emotional video, the method reduces the time complexity and improves the accuracy of the emotion recognition.