The present invention relates to a multi-modal fusion gesture recognition method based on the color and depth information. The method comprises the steps of 1) simultaneously shooting a gesture by a color camera and a depth camera to obtain the original data of the color and the depth of the gesture; 2) based on the gaussian smoothing method, subjecting the original data of the color and the depth of the gesture to filtering to obtain the smooth data of the color and the depth of the gesture; 3) adopting the skin color and the depth as the priori knowledge, respectively segmenting the smooth data of the color and the depth to obtain a skin color segmentation region and a depth segmentation region; 4) fusing the pixel layers of the skin color segmentation region and the depth segmentation region based on dynamic weights so as to form a fused segmentation map; 5) extracting the features of the fused segmentation map by using a feature descriptor and forming a feature map; 6 ) encoding the feature map to form gesture codes; 7) recognizing the gesture codes to realize the gesture recognition. Compared with the prior art, the above method is higher in gesture recognition accuracy.