The invention discloses a gesture tracking recognition joint strategy method oriented to deskVR, At first, that hand gesture public data set ICVL is train offline by the convolution neural network, get a good robustness, models with high accuracy and fast recognition speed, and by testing the balance point Rate of accuracy and speed of the combined method with synthetic data, Then the depth imagesare captured by the depth camera in real time. After a series of preprocessing, the images are transferred into the gesture recognition method based on the fast model or the five-region convolution neural network model according to the proportion of the hand images, and the 3D information of the recognized joints is returned, and then the 3D human body model is mapped. The recognition method of the invention has good interactive function under the deskVR application environment, and performs well in recognition rate, operation speed, universality, accuracy and the like.