A system for grouping image annotations is disclosed. A display unit (1) is arranged for displaying a plurality of medical images relating to a subject (5) comprising a representation of an anatomical object. An annotation unit (2) is arranged for receiving information regarding input of a plurality of annotations (3) of the plurality of medical images via a user interface (4), wherein each annotation comprises geometric information relative to an image of the plurality of medical images. A grouping unit (4) is arranged for detecting a group of annotations among the plurality of annotations, based on the geometric information and a set of rules that define general properties of geometric relationships of annotations (3) that belong to the same object. The detecting of the group of annotations is further based on a geometric relationship between the plurality of medical images.