A method generates a three-dimensional, bi-linear, illumination model for arbitrary faces. A large number of images are acquired of many different faces. For each face, multiple images are acquired with varying poses and varying illumination. A three-mode singular value decomposition is applied to the images to determine parameters of the model. The model can be fit to a probe image of an unknown face. Then, the model can be compared with models of a gallery of images of unknown faces to recognize the face in the probe image.