公开(公告)号：US20220092429A1

公开(公告)日：2022-03-24

申请号：US17481160

申请日：2021-09-21

Applicant: Google LLC

Inventor： Luke Shekerjian Metz ,  Niruban Maheswaranathan ,  Christian Daniel Freeman ,  Benjamin Poole ,  Jascha Narain Sohl-Dickstein

IPC: G06N3/08

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a neural network. One of the methods includes performing, using a plurality of training examples, a training step to obtain respective gradients of a loss function with respect to each of the parameters in the parameter tensors; obtaining a validation loss for a plurality of validation examples that are different from the plurality of training examples generating an optimizer input from at least the respective gradients and the validation loss; processing the optimizer input using an optimizer neural network to generate an output defining a respective update for each of the parameters in the parameter tensors of the neural network; and for each of the parameters in the parameter tensors, applying the respective update to a current value of the parameter to generate an updated value for the parameter.

公开(公告)号：US20230059708A1

公开(公告)日：2023-02-23

申请号：US17797966

申请日：2021-02-08

Applicant: Google LLC

Inventor： Luke Shekerjian Metz ,  Ruoxi Sun ,  Christian Daniel Freeman ,  Benjamin Michael Poole ,  Niru Maheswaranathan ,  Jascha Narain Sohl-Dickstein

IPC: G06N3/08

Abstract: The present disclosure provides a computer-implemented method for determining an optimized list of sets of hyperparameter values for application to an additional machine learning task. The method includes obtaining data describing a plurality of different machine learning tasks. The method includes obtaining a plurality of candidate sets of hyperparameter values. The method includes determining an ordered list of sets of hyperparameters selected from the plurality of candidate sets of hyperparameter values, wherein the ordered list of sets of hyperparameters minimizes an aggregate loss over the plurality of different machine learning tasks. The method includes storing the ordered list of sets of hyperparameters for use in training an additional machine learning model to perform an additional machine learning task.

公开(公告)号：US20220391706A1

公开(公告)日：2022-12-08

申请号：US17831338

申请日：2022-06-02

Applicant: Google LLC

Inventor： Luke Shekerjian Metz ,  Christian Daniel Freeman ,  Jascha Narain Sohl-Dickstein ,  Niruban Maheswaranathan ,  James Michael Harrison

IPC: G06N3/08

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training neural networks using learned optimizers. One of method is for training a neural network layer comprising a plurality of network parameters having a plurality of dimensions each having a plurality of indices, the method comprising: maintaining a set of values corresponding to respective sets of indices of each dimension, each value representing a measure of central tendency of past gradients of the network parameters having an index in the dimension that is in the set of indices; performing a training step to obtain a new gradient for each network parameter; updating each set of values using the new gradients; and for each network parameter: generating an input from the updated sets of values; processing the input using an optimizer neural network to generate an output defining an update for the network parameter; and applying the update.