A system and method for categorizing color palettes are provided. A taxonomy for a particular type of achromatic entity, such as emotions, is generated and populated with a list of terms. Natural language terms extracted from annotated color palettes are associated with the corresponding achromatic entity categories in the taxonomy. Features are extracted from the color palettes. The features, together with the category information, are used to train a classifier which is then able to assign achromatic entities to palettes outside the training set, based on extracted features.