An image search is executed. The search results are presented as a collection of thumbnail images organized according to an initial ranking. As the user browses the thumbnail images, he/she encounters a thumbnail image of interest. Selecting this thumbnail causes an enlarged preview image to be displayed. The user can manipulate the enlarged preview image with panning and zooming operations, thereby enabling the user to more closely inspect portions of the preview image which attract the user's interest. These interactions with the search results, which include the initial selection of the thumbnail image and the subsequent manipulation of the enlarged preview image, provide insight into the user's interest. Once the user has interacted with a threshold quantity of search results, the collection of thumbnail images is filtered and reorganized to more prominently position those search results that correspond to the user's interest, as gleaned from analyzing the aforementioned user interactions.