In order to compare high-dimensional, multi-modal data for a patient to data for other patients, deep learning is used to encode original, multi-modal data for a patient into a compact signature. The compact signature is compared to predetermined compact signatures generated for other patients, and similar predetermined compact signatures are identified based on the comparison. A clinical outcome may be predicted based on the similar predetermined compact signatures that are identified.