In one embodiment, a device receives, from a plurality of training nodes that train a set of machine learning models using local training datasets, bias metrics associated with those machine learning models for each feature of the local training datasets. The device generates aggregated machine learning models over time that aggregate the machine learning models trained by the plurality of training nodes. The device constructs, based on the bias metrics, bias lineages for the aggregated machine learning models. The device provides, based on the bias lineages, a bias lineage for a particular one of the aggregated machine learning models for display.