A computer-implemented system is described for identifying and retrieving similar radiology images to a query image. The system includes one or more fetchers receiving the query image and retrieving a set of candidate similar radiology images from a data store. One or more scorers receive the query image and the set of candidate similar radiology images and generate a similarity score between the query image and each candidate image. A pooler receives the similarity scores from the one or more scorers, ranks the candidate images, and returns a list of the candidate images reflecting the ranking. The scorers implement a modelling technique to generate the similarity score capturing a plurality of similarity attributes of the query image and the set of candidate similar radiology images and annotations associated therewith. For example, the similarity attributes could be patient, diagnostic and/or visual similarity, and the modelling techniques could be triplet loss, classification loss, regression loss and object detection loss.