A color palette for an image is obtained, which is a set of different colors that are determined to be the dominant colors in the image. The color palette can be displayed to a user, and a user input in the form of an adverbial expression to change the color palette is received. The adverbial expression can indicate various different changes to the color palette, such as more or less of a particular color, making a color brighter or darker, making all colors in the color palette shallower or deeper, increasing or reducing the contrast of colors in the color palette, and so forth. The adverbial expression can be transformed into a new color palette and a color based image search can be initiated to identify images based on the colors in the new color palette, or an image can be modified based on the adverbial expression.