In an exemplary embodiment, an on-board camera is used as a controller for a hand-held game device. Control information is provided to a game application via video captured by the camera. As the hand-held device is moved, features of the captured video are tracked. The relative motion between the tracked features and the hand-held device is translated into control information for the game application. Aspects of the game application are rendered on a display of the hand-held game device in accordance with the control information. For example, from a third person perspective, viewing a game character from overhead, rotation of the hand-held game device can be translated into rotation of the character. From a first person perspective, seeing the game through the eyes of the game character, rotation of the hand-held game device can be translated into observing various challenges coming from different directions.