This document describes techniques and apparatuses for constructing three dimensional (3D) surfaces for multi-colored objects. In some aspects, these techniques determine, from a color image and coarse depth information, an illumination model and albedo for a multi-color object. The coarse depth information may then be refined based on the illumination model and combined with the albedo to provide a 3D surface of the multi-color object.