Taking a set of unlabeled images of a collection of objects acquired under different imaging conditions, and decomposing the set into disjoint subsets corresponding to individual objects requires clustering. Appearance-based methods for clustering a set of images of 3-D objects acquired under varying illumination conditions can be based on the concept of illumination cones. A clustering problem is equivalent to finding convex polyhedral cones in the high-dimensional image space. To efficiently determine the conic structures hidden in the image data, the concept of conic affinity can be used which measures the likelihood of a pair of images belonging to the same underlying polyhedral cone. Other algorithms can be based on affinity measure based on image gradient comparisons operating directly on the image gradients by comparing the magnitudes and orientations of the image gradient.