A cost-effective and computationally efficient hand gesture recognition system for detecting and/or tracking a face region and/or a hand region in a series of images. A skin segmentation model is updated with skin pixel information from the face and iteratively applied to the pixels in the hand region, to more accurately identify the pixels in the hand region given current lighting conditions around the image. Shape features are then extracted from the image, and based on the shape features, a hand gesture is identified in the image. The identified hand gesture may be used to generate a command signal to control the operation of an application or system.