This disclosure describes various exemplary user interfaces, methods, and computer program products for the interactively ranking image search results refinement method using a color layout. The method includes receiving a text query for an image search, presenting image search results in a structured presentation based on the text query and information from an interest color layout. The process creates image search results that may be selected by the user based on color selection palettes or color layout specification schemes. Then the process ranks the image search results by sorting the results according to similarity scores between color layouts from the image search results and the interest color layout from a user based on the color selection palettes and the color layout specification schemes.