Methods and systems for gesture-based control of a device are described. A virtual gesture-space is determined in a received input frame. The virtual gesture-space is associated with a primary user from a ranked user list of users. The received input frame is processed in only the virtual gesture-space, to detect and track a hand. Using a hand bounding box generated by detecting and tracking the hand, gesture classification is performed to determine a gesture input associated with the hand. A command input associated with the determined gesture input is processed. The device may be a smart television, a smart phone, a tablet, etc.