Disclosed herein are system, method, and computer program product embodiments for providing an image search training system. An embodiment operates by determining a query image on which to train an image search system, and a positive image visually similar to the query image. A set of negative images from a plurality of negative images visually dissimilar to the query image are selected, where the selected set of negative images includes both a first negative image and a second negative image. A first similarity measure between the first negative image and the positive image, and a second similarity measure between the second negative image and the positive image are calculated. The first negative image is selected based on the first similarity measure being less than the second similarity measure. The query image, the positive image, and the first negative image are provided to the image search system for training.