Provided is a method of synthesizing 3D joint data based on a multi-view RGB-D camera. The method of synthesizing 3D joint data includes: converting joint data collected from a depth camera of each of a plurality of RGB-D cameras from a depth camera coordinate system of each of the RGB-D cameras to a color camera coordinate system of each of the RGB-D cameras; calculating a confidence level of the joint data converted to the color camera coordinate system using a 3D joint recognition algorithm based on the joint data converted to the color camera coordinate system; applying a rotation matrix and a translation vector to the joint data and converting the joint data to a predetermined reference coordinate system; and obtaining a weighted-average of the joint data converted to the reference coordinate system using a weight calculated based on the confidence level to synthesize the joint data.