The present disclosure is directed to a deep learning system for differential diagnoses of skin diseases. In particular, the system performs a method that can include obtaining a plurality of images that respectively depict a portion of a patient's skin. The method can include determining, using a machine-learned skin condition classification model, a plurality of embeddings respectively for the plurality of images. The method can include combining the plurality of embeddings to obtain a unified representation associated with the portion of the patient's skin. The method can include determining, using the machine-learned skin condition classification model, a skin condition classification for the portion of the patients skin, the skin condition classification produced by the machine-learned skin condition classification model by processing the unified representation, wherein the skin condition classification identifies one or more skin conditions selected from a plurality of potential skin conditions.