A computing system may determine a head pose and a wrist pose of a user based on sensor data captured by a head-mounted device. The computing system may further receive an image captured by a camera of the head-mounted device that includes at least portion of the body of the user. Using this image, the computing system may determine one or more regions in the image that correspond to the body of the user, and determine, based on these regions and a camera pose of the camera, a three-dimensional volume constraint in which an elbow of the user is likely to be located when the image was captured. From this information, the computing system may utilize the head pose, the wrist pose, and the three-dimensional volume constraint to infer a body pose of the user that includes at least an inferred elbow pose associated with the user.