Various techniques are disclosed for controlling a wager-based game played at a gaming system. In one embodiment the gaming system may include a handheld device operable to communicate with a gaming controller. In one embodiment, the handheld device may include a gesture input interface device operable to detect movements gestures associated with one or more persons. The gaming system may further include a gesture interpretation component operable to identify selected movements or gestures detected by the gesture input interface device, and operable to generate gesture interpretation information relating to interpretation of the selected movements or gestures. In at least one embodiment, the gaming system may be operable to control a wager-based game played at the gaming system; automatically detect a gesture by a player participating in a game session at the gaming system; interpret the gesture with respect to a set of criteria; identify at least one action to be initiated in response to the gesture interpretation; and initiate the at least one action. In at least one embodiment, information relating to the gesture and/or information relating to the gesture interpretation may be recorded as part of the game history associated with the game session.