A system for automatically classifying an object of a target image is provided. A classification system provides a collection of classified images along with a classification of the dominant object of the image. The classification system attempts to classify the object of a target image based on similarity of the target image to the classified images. To classify a target image, the classification system identifies the classified images of the collection that are most similar to the target image based on similarity between salient points of the target image and the classified images. The classification system selects a classification associated with the classified images that are most similar to the target image as a classification for the object of the target image.