Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating a query image model. In one aspect, a method includes receiving a set of images determined to be responsive to a query and ranked according to a first order; determining a positive image signature from a first subset of images selected from images ranked highest in the first order, determining a negative image signature from a second subset of images selected from images ranked lowest in the first order, determining a query image signature for the query based on a difference of the positive image signature and the negative image signature; and applying the query image signature to each image in the set of images to rank the images according to a second order that is different from the first order.