Systems and methods for processing digital medical images to infer metadata from those images are disclosed. In some aspects, digital medical images may be processed to infer metadata by receiving a plurality of digital medical images, receiving a prompt, the prompt being a request for a specific type of metadata to be inferred from the plurality of digital medical images, determining, using a trained foundation model, at least one feature descriptor from the plurality of digital medical images based on the prompt, and providing for output the at least one feature descriptor for each of the plurality of digital medical images.