In one embodiment, a device determines that a machine learning model is to be trained by a plurality of devices in a network. A set of training devices are identified from among the plurality of devices to train the model, with each of the training devices having a local set of training data. An instruction is then sent to each of the training devices that is configured to cause a training device to receive model parameters from a first training device in the set, use the parameters with at least a portion of the local set of training data to generate new model parameters, and forward the new model parameters to a second training device in the set. Model parameters from the training devices are also received that have been trained using a global set of training data that includes the local sets of training data on the training devices.