公开(公告)号：US20210158156A1

公开(公告)日：2021-05-27

申请号：US17025418

申请日：2020-09-18

Applicant: Google LLC

Inventor： Gil Shamir ,  Lorenzo Coviello

IPC: G06N3/08 ,  G06N3/04

Abstract: Systems and methods can improve the reproducibility of neural networks by distilling from ensembles. In particular, aspects of the present disclosure are directed to a training scheme that utilizes a combination of an ensemble of neural networks and a single, “wide” neural network that is more powerful (e.g., exhibits a greater accuracy) than the ensemble. Specifically, the output of the ensemble can be distilled into the single neural network during training of the single neural network. After training, the single neural network can be deployed to generate inferences. In such fashion, the single neural model can provide a superior prediction accuracy while, during training, the ensemble can serve to influence the single neural network to be more reproducible. In addition, an additional single wide tower can be added to generate another output, that can be distilled to the single neural network, to further improve its accuracy.