A system and method for real-time texture mapping for an augmented reality system are described. A viewing device includes an optical sensor to capture an image of a real-world object. A texture extraction module extracts a texture of the image of the real-world object. A recognition module identifies the real-world object based on the captured image. A texture mapping module retrieves a virtual object corresponding to the identified real-world object, maps the texture to the virtual object, dynamically updates the texture to the virtual object in real time, and generates a visualization of the virtual object in a display of the viewing device.