A method of modeling a human subject includes receiving from a depth camera a depth map of a scene including the human subject. The human subject is modeled with a virtual skeleton including a plurality of virtual joints. Each virtual joint is defined with a three-dimensional position. Furthermore, each of the plurality of virtual joints is further defined with three orthonormal vectors. The three orthonormal vectors for each virtual joint provide an orientation of that virtual joint at the three-dimensional position defined for that virtual joint.