An apparatus and method for natural hand processing for virtual reality. For example, one embodiment of an apparatus comprises: graphics processing circuitry to render left and right scenes responsive to a virtual reality application, the left and right scenes to be displayed on left and right displays, respectively, of a virtual reality device; and a natural hand processing pipeline to project a representation of a user's hands into the left and right scenes to allow the user to interact with one or more objects in the scenes, the natural hand processing pipeline comprising: a depth/mask data smoothing and reconstruction module to receive noisy depth and mask data associated with images of the user's hands and to enhance the noisy depth data and mask data to generate enhanced depth and mask data; a hand mesh generator to generate a hand mesh using the enhanced depth and mask data; and a mesh data projector to project the hand mesh into the left and right scenes.