公开(公告)号：US20240428071A1

公开(公告)日：2024-12-26

申请号：US18823611

申请日：2024-09-03

Applicant: Google LLC

Inventor： David Richard So ,  Quoc V. Le ,  Hanxiao Liu ,  Wojciech Andrzej Manke ,  Zihang Dai ,  Noam M. Shazeer

IPC: G06N3/08

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for performing a machine learning task on a network input to generate a network output. One of the systems includes an attention neural network configured to perform the machine learning task. The attention neural network includes one or more attentions layers that each include a squared ReLU activation layer, a depth-wise convolution layer, or both.

公开(公告)号：US20220383119A1

公开(公告)日：2022-12-01

申请号：US17827362

申请日：2022-05-27

Applicant: Google LLC

Inventor： David Richard So ,  Quoc V. Le, Jr. ,  Hanxiao Liu ,  Wojciech Andrzej Manke ,  Zihang Dai ,  Noam M. Shazeer

IPC: G06N3/08

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for performing a machine learning task on a network input to generate a network output. One of the systems includes an attention neural network configured to perform the machine learning task. The attention neural network includes one or more attentions layers that each include a squared ReLU activation layer, a depth-wise convolution layer, or both.