A system determines what regions in a scene captured by a camera include highly reflective (specular) surfaces. The system illuminates the scene by outputting images to a display screen such as a television screen so as to illuminate the scene from different angles. Images of the scene are captured by the camera under the different illuminations. The images are analyzed to determine which regions in the scene exhibit changes in reflected luminance that correspond to the changes in illumination, indicating a specular surface. Regions containing diffuse surfaces are also identified based on such regions exhibiting a reflected luminance that is substantially independent of the changes in illumination.