公开(公告)号：US20210383223A1

公开(公告)日：2021-12-09

申请号：US17337834

申请日：2021-06-03

Applicant: Google LLC

Inventor： Mingxing Tan ,  Xuanyi Dong ,  Wei Yu ,  Quoc V. Le ,  Daiyi Peng

IPC: G06N3/08 ,  G06N3/04

Abstract: The present disclosure provides a differentiable joint hyper-parameter and architecture search approach, with some implementations including the idea of discretizing the continuous space into a linear combination of multiple categorical basis. One example element of the proposed approach is the use of weight sharing across all architecture- and hyper-parameters which enables it to search efficiently over the large joint search space. Experimental results on MobileNet/ResNet/EfficientNet/BERT show that the proposed systems significantly improve the accuracy by up to 2% on ImageNet and the F1 by up to 0.4 on SQuAD, with search cost comparable to training a single model. Compared to other AutoML methods, such as random search or Bayesian method, the proposed techniques can achieve better accuracy with 10× less compute cost.

公开(公告)号：US20230144138A1

公开(公告)日：2023-05-11

申请号：US17905196

申请日：2021-06-04

Applicant: Google LLC

Inventor： Daiyi Peng ,  Yifeng Lu ,  Quoc V. Le

IPC: G06N20/00 ,  G06N5/01

CPC classification number: G06N20/00 ,  G06N5/01

Abstract: A method for searching for an output machine learning (ML) algorithm to perform an ML task is described. The method comprising: receiving data specifying an input ML algorithm; receiving data specifying a search algorithm that searches for candidate ML algorithms and an evaluation function that evaluates the performance of candidate ML algorithms; generating data representing a symbolic tree from the input ML algorithm; generating data representing a hyper symbolic tree from the symbolic tree; searching an algorithm search space that defines a set of possible concrete symbolic trees from the hyper symbolic tree for candidate ML algorithms and training the candidate ML algorithms to determine a respective performance metric for each candidate ML algorithm; and selecting one or more trained candidate ML algorithms among the trained candidate ML algorithms based on the determined performance metrics.

公开(公告)号：US20220391687A1

公开(公告)日：2022-12-08

申请号：US17338093

申请日：2021-06-03

Applicant: Google LLC

Inventor： John Dalton Co-Reyes ,  Yingjie Miao ,  Daiyi Peng ,  Sergey Vladimir Levine ,  Quoc V. Le ,  Honglak Lee ,  Aleksandra Faust

IPC: G06N3/08 ,  G06F11/34 ,  G06F16/901

Abstract: Methods, computer systems, and apparatus, including computer programs encoded on computer storage media, for generating and searching reinforcement learning algorithms. In some implementations, a computer-implemented system generates a sequence of candidate reinforcement learning algorithms. Each candidate reinforcement learning algorithm in the sequence is configured to receive an input environment state characterizing a state of an environment and to generate an output that specifies an action to be performed by an agent interacting with the environment. For each candidate reinforcement learning algorithm in the sequence, the system performs a performance evaluation for a set of a plurality of training environments. For each training environment, the system adjusts a set of environment-specific parameters of the candidate reinforcement learning algorithm by performing training of the candidate reinforcement learning algorithm to control a corresponding agent in the training environment. The system generates an environment-specific performance metric for the candidate reinforcement learning algorithm that measures a performance of the candidate reinforcement learning algorithm in controlling the corresponding agent in the training environment as a result of the training. After performing training in the set of training environments, the system generates a summary performance metric for the candidate reinforcement learning algorithm by combining the environment-specific performance metrics generated for the set of training environments. After evaluating each of the candidate reinforcement learning algorithms in the sequence, the system selects one or more output reinforcement learning algorithms from the sequence based on the summary performance metrics of the candidate reinforcement learning algorithms.

公开(公告)号：US20240289605A1

公开(公告)日：2024-08-29

申请号：US18173347

申请日：2023-02-23

Applicant: Google LLC

Inventor： Lav Rai ,  Xiang Xu ,  Yen-Min Hsu ,  Bo Wu ,  Daiyi Peng

IPC: G06N3/08

CPC classification number: G06N3/08

Abstract: Aspects of the disclosure are directed to proxy task design tools that automatically find proxy tasks, such as optimal proxy tasks, for neural architecture searches. The proxy task design tools can include one or more tools to search for an optimal proxy task having the lowest neural architecture search cost while meeting a minimum correlation requirement threshold after being provided with a proxy task search space definition. The proxy task design tools can further include one or more tools to select candidate models for computing correlation scores of proxy tasks as well as one or more tools to measure variance of a model. The proxy task design tools can minimize time and effort involved in designing the proxy task.

公开(公告)号：US20240112027A1

公开(公告)日：2024-04-04

申请号：US18477546

申请日：2023-09-28

Applicant: Google LLC

Inventor： Yanqi Zhou ,  Yanping Huang ,  Yifeng Lu ,  Andrew M. Dai ,  Siamak Shakeri ,  Zhifeng Chen ,  James Laudon ,  Quoc V. Le ,  Da Huang ,  Nan Du ,  David Richard So ,  Daiyi Peng ,  Yingwei Cui ,  Jeffrey Adgate Dean ,  Chang Lan

IPC: G06N3/08

CPC classification number: G06N3/08

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing neural architecture search for machine learning models. In one aspect, a method comprises receiving training data for a machine learning, generating a plurality of candidate neural networks for performing the machine learning task, wherein each candidate neural network comprises a plurality of instances of a layer block composed of a plurality of layers, for each candidate neural network, selecting a respective type for each of the plurality of layers from a set of layer types that comprises, training the candidate neural network and evaluating performance scores for the trained candidate neural networks as applied to the machine learning task, and determining a final neural network for performing the machine learning task based at least on the performance scores for the candidate neural networks.

公开(公告)号：US20240005129A1

公开(公告)日：2024-01-04

申请号：US18029849

申请日：2021-10-01

Applicant: Google LLC

Inventor： Yanqi Zhou ,  Amir Yazdanbakhsh ,  Berkin Akin ,  Daiyi Peng ,  Yuxiong Zhu ,  Mingxing Tan ,  Xuanyi Dong

IPC: G06N3/045 ,  G06N3/092 ,  G06N3/063 ,  G06N3/044 ,  G06N3/0464

CPC classification number: G06N3/045 ,  G06N3/092 ,  G06N3/0464 ,  G06N3/044 ,  G06N3/063

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for jointly determining neural network architectures and hardware accelerator architectures. In one aspect, a method includes: generating, using a controller policy, a batch of one or more output sequences, each output sequence in the batch defining a respective architecture of a child neural network and a respective architecture of a hardware accelerator; for each output sequence in the batch: training a respective instance of the child neural network having the architecture defined by the output sequence; evaluating a network performance of the trained instance of the child neural; and evaluating an accelerator performance of a respective instance of the hardware accelerator having the architecture defined by the output sequence to determine an accelerator performance metric for the instance of the hardware accelerator; and using the network performance metrics and the accelerator performance metrics to adjust the controller policy.

公开(公告)号：US20220019856A1

公开(公告)日：2022-01-20

申请号：US17377142

申请日：2021-07-15

Applicant: Google LLC

Inventor： Jaehoon Lee ,  Daiyi Peng ,  Yuan Cao ,  Jascha Narain Sohl-Dickstein ,  Daniel Sung-Joon Park

IPC: G06K9/62 ,  G06N3/08

Abstract: A method for predicting performance of a neural network (NN) is described. The method includes receiving a training data set having a set of training samples; receiving a validation data set having a set of validation pairs; initializing (i) a validation-training kernel matrix representing similarities of the validation inputs in the validation data set and the training inputs in the training data set and (ii) a training-training kernel matrix representing similarities across the training inputs within the training data set; generating a final updated validation-training kernel matrix and a final updated training-training kernel matrix; performing the following operations at least once: generating predicted validation outputs for the validation inputs, and updating an accuracy score of the NN based on the predicted validation outputs and the validation outputs; and outputting the updated accuracy score as a final accuracy score representing performance of the NN.