公开(公告)号：US20210383237A1

公开(公告)日：2021-12-09

申请号：US17337812

申请日：2021-06-03

Applicant: Google LLC

Inventor： Mingxing Tan ,  Cihang Xie ,  Boqing Gong ,  Quoc V. Le

IPC: G06N3/08 ,  G06N3/04

Abstract: Generally, the present disclosure is directed to the training of robust neural network models by using smooth activation functions. Systems and methods according to the present disclosure may generate and/or train neural network models with improved robustness without incurring a substantial accuracy penalty and/or increased computational cost, or without any such penalty at all. For instance, in some examples, the accuracy may improve. A smooth activation function may replace an original activation function in a machine-learned model when backpropagating a loss function through the model. Optionally, one activation function may be used in the model at inference time, and a replacement activation function may be used when backpropagating a loss function through the model. The replacement activation function may be used to update learnable parameters of the model and/or to generate adversarial examples for training the model.