A method and system for automatic semantics driven registration of medical images is disclosed. Anatomic landmarks and organs are detected in a first image and a second image. Pathologies are also detected in the first image and the second image. Semantic information is automatically extracted from text-based documents associated with the first and second images, and the second image is registered to the first image based the detected anatomic landmarks, organs, and pathologies, and the extracted semantic information.