公开(公告)号：US20220398500A1

公开(公告)日：2022-12-15

申请号：US17332893

申请日：2021-05-27

Applicant: Google LLC

Inventor： Karan Singhal ,  Hakim Sidahmed, JR. ,  Zachary A. Garrett ,  Shanshan Wu ,  John Keith Rush ,  Sushant Prakash

IPC: G06N20/20

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a machine learning model having a set of local model parameters and a set of global model parameters under a partially local federated learning framework. One of the methods include maintaining local data and data defining the local model parameters; receiving data defining current values of the global model parameters; determining, based on the local data, the local model parameters, and the current values of the global model parameters, current values of the local model parameters; determining, based on the local data, the current values of the local model parameters, and the current values of the global model parameters, updated values of the global model parameters; generating, based on the updated values of the global model parameters, parameter update data defining an update to the global model parameters; and transmitting the parameter update data.

公开(公告)号：US12271810B2

公开(公告)日：2025-04-08

申请号：US17100253

申请日：2020-11-20

Applicant: Google LLC

Inventor： Sashank Jakkam Reddi ,  Sanjiv Kumar ,  Manzil Zaheer ,  Zachary Burr Charles ,  Zachary Alan Garrett ,  John Keith Rush ,  Jakub Konecny ,  Hugh Brendan McMahan

IPC: G06N3/08 ,  G06N3/04 ,  G06N3/045

Abstract: A computing system and method can be used to implement a version of federated learning (FL) that incorporates adaptivity (e.g., leverages an adaptive learning rate). In particular, the present disclosure provides a general optimization framework in which (1) clients perform multiple epochs of training using a client optimizer to minimize loss on their local data and (2) a server system updates its global model by applying a gradient-based server optimizer to the average of the clients' model updates. This framework can seamlessly incorporate adaptivity by using adaptive optimizers as client and/or server optimizers. Building upon this general framework, the present disclosure also provides example specific adaptive optimization techniques for FL which use per-coordinate methods as server optimizers. By focusing on adaptive server optimization, the use of adaptive learning rates is enabled without increase in client storage or communication costs and compatibility with cross-device FL can be ensured.