The technology provides for a system for determining a gesture provided by a user. In this regard, one or more processors of the system may receive image data from one or more visual sensors of the system capturing a motion of the user, and may receive motion data from one or more wearable computing devices worn by the user. The one or more processors may recognize, based on the image data, a portion of the user's body that corresponds to a gesture to perform a command. The one or more processors may also determine one or more correlations between the image data and the received motion data. Based on the recognized portion of the user's body and the one or more correlations between the image data and the received motion data, the one or more processors may detect the gesture.