A hierarchical multi-object active appearance model (AAM) framework is disclosed for processing image data, such as localizer or scout image data. In accordance with this approach, a hierarchical arrangement of models (e.g., a model pyramid) maybe employed where a global or parent model that encodes relationships across multiple co-located structures is used to obtain an initial, coarse fit. Subsequent processing by child sub-models add more detail and flexibility to the overall fit.