The invention discloses a hand motion recognition method based on a depth image and a color image. The method comprises the following steps: taking 36 types of gestures of an ASL sign language libraryas templates, obtaining gesture data through a Kinect sensor, and building a gesture database under the depth and color backgrounds; a regression-based target detection algorithm SSD is used as a research basis; under a Tensorflow deep learning framework, transfer learning is carried out on a selected target detection model by utilizing a gesture database self-built based on color and depth backgrounds respectively to obtain two types of network models capable of carrying out recognition detection on hand movement under the depth and color backgrounds. A hand motion recognition detection network framework with detection results fused under depth and color backgrounds is utilized, a non-maximum suppression algorithm is improved, and finally the effectiveness of hand motion recognition detection of the proposed network framework is obtained. According to the invention, the problems of missing detection and false detection of the target are avoided, the gesture recognition rate is improved, and single-hand recognition and double-hand recognition can be realized.