The invention relates to a matching method of body images and a video analysis retrieval system. A first video partial decoding to the video to be analyzed is carried out to obtain the needed detection information, then a motion detection is operated and a second video partial decoding only to the moving images is carried out; the information of brightness and color of the images is obtained; thebody information is obtained by detecting and tracking the body and stored into a data-base. The inquiring information for an inquirer is inputted when an inquiry is needed; and via accessing the bodyinformation in the data-base and adopting the body matching algorithm to obtain a matching result which matches with the inquiring information of the inquirer, the matching result is outputted and displayed. The method and the system quicken the processing speed and improve the inquiring precision.