The present invention is directed to a method for populating a database with a set of images of an anatomical structure. The database is used to perform appearance matching in image pairs of the anatomical structure. A set of image pairs of anatomical structures is received, where each image pair is annotated with a plurality of location-sensitive regions that identify a particular aspect of the anatomical structure. Weak learners are iteratively selected and an image patch is identified. A boosting process is used to identify a strong classifier based on responses to the weak learners applied to the identified image patch for each image pair. The responses comprise a feature response and a location response associated with the image patch. Positive and negative image pairs are generated. The positive and negative image pairs are used to learn a similarity function. The learned similarity function and iteratively selected weak learners are stored in the database.