A method models a three-dimensional object with a compressed surface reflectance field. Images of the object are acquired with multiple cameras for multiple viewpoints under different lighting conditions. The images are stored in a matrix Mr representing a surface reflectance field for the three-dimensional object. The matrix Mr is factorized into principle components pck and coefficients cfk. The principal components pck are stored in a matrix Mpc. The matrix Mpc is then factorized into principle components pcm and coefficients cfk which can be stored for each vertex V of a model of the three-dimensional object. The corresponding values of the principle components pcm, coefficients cfk, and coefficients cfk, respectively represent a compression of a surface map, which can be rendered from arbitrary viewpoints and under arbitrary lighting conditions.