A system (100) for detecting and labeling structures of interest includes a current patient study database (102) containing a current patient study (200) with clinical contextual information (706), a statistical model patient report database (104) containing at least one or more prior patient documents containing clinical contextual information (706), an image metadata processing engine (118) configured to extract metadata for preparing an input for an anatomical structure classifier (608), a natural language processing engine (120) configured to extract clinical context information (706) from the prior patient documents, an anatomical structure detection and labeling engine (718) or processor (112), and a display device (108) configured to display findings from the current patient study. The anatomical structure detection and labeling engine (718) or processor (112) is configured to identify and label one or more structures of interest (716) from the extracted metadata and clinical context information (706) and aggregate series level data.