公开(公告)号：US11893483B2

公开(公告)日：2024-02-06

申请号：US16988547

申请日：2020-08-07

Applicant: Google LLC

Inventor： Noam M. Shazeer ,  Aidan Nicholas Gomez ,  Lukasz Mieczyslaw Kaiser ,  Jakob D. Uszkoreit ,  Llion Owen Jones ,  Niki J. Parmar ,  Illia Polosukhin ,  Ashish Teku Vaswani

IPC: G06N20/00 ,  G06N3/08 ,  G06N3/045 ,  G06N3/04

CPC classification number: G06N3/08 ,  G06N3/04 ,  G06N3/045 ,  G06N20/00

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output sequence from an input sequence. In one aspect, one of the systems includes an encoder neural network configured to receive the input sequence and generate encoded representations of the network inputs, the encoder neural network comprising a sequence of one or more encoder subnetworks, each encoder subnetwork configured to receive a respective encoder subnetwork input for each of the input positions and to generate a respective subnetwork output for each of the input positions, and each encoder subnetwork comprising: an encoder self-attention sub-layer that is configured to receive the subnetwork input for each of the input positions and, for each particular input position in the input order: apply an attention mechanism over the encoder subnetwork inputs using one or more queries derived from the encoder subnetwork input at the particular input position.

公开(公告)号：US11113602B2

公开(公告)日：2021-09-07

申请号：US16932422

申请日：2020-07-17

Applicant: Google LLC

Inventor： Noam M. Shazeer ,  Aidan Nicholas Gomez ,  Lukasz Mieczyslaw Kaiser ,  Jakob D. Uszkoreit ,  Llion Owen Jones ,  Niki J. Parmar ,  Illia Polosukhin ,  Ashish Teku Vaswani

IPC: G06N3/04 ,  G06N3/08

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output sequence from an input sequence. In one aspect, one of the systems includes an encoder neural network configured to receive the input sequence and generate encoded representations of the network inputs, the encoder neural network comprising a sequence of one or more encoder subnetworks, each encoder subnetwork configured to receive a respective encoder subnetwork input for each of the input positions and to generate a respective subnetwork output for each of the input positions, and each encoder subnetwork comprising: an encoder self-attention sub-layer that is configured to receive the subnetwork input for each of the input positions and, for each particular input position in the input order: apply an attention mechanism over the encoder subnetwork inputs using one or more queries derived from the encoder subnetwork input at the particular input position.

公开(公告)号：US12217173B2

公开(公告)日：2025-02-04

申请号：US17467096

申请日：2021-09-03

Applicant: Google LLC

Inventor： Noam M. Shazeer ,  Aidan Nicholas Gomez ,  Lukasz Mieczyslaw Kaiser ,  Jakob D. Uszkoreit ,  Llion Owen Jones ,  Niki J. Parmar ,  Illia Polosukhin ,  Ashish Teku Vaswani

IPC: G06N20/00 ,  G06N3/04 ,  G06N3/045 ,  G06N3/08

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output sequence from an input sequence. In one aspect, one of the systems includes an encoder neural network configured to receive the input sequence and generate encoded representations of the network inputs, the encoder neural network comprising a sequence of one or more encoder subnetworks, each encoder subnetwork configured to receive a respective encoder subnetwork input for each of the input positions and to generate a respective subnetwork output for each of the input positions, and each encoder subnetwork comprising: an encoder self-attention sub-layer that is configured to receive the subnetwork input for each of the input positions and, for each particular input position in the input order: apply an attention mechanism over the encoder subnetwork inputs using one or more queries derived from the encoder subnetwork input at the particular input position.

公开(公告)号：US20220083746A1

公开(公告)日：2022-03-17

申请号：US17459041

申请日：2021-08-27

Applicant: Google LLC

Inventor： Zhifeng Chen ,  Macduff Richard Hughes ,  Yonghui Wu ,  Michael Schuster ,  Xu Chen ,  Llion Owen Jones ,  Niki J. Parmar ,  George Foster ,  Orhan Firat ,  Ankur Bapna ,  Wolfgang Macherey ,  Melvin Jose Johnson Premkumar

IPC: G06F40/58 ,  G06N3/08

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for machine translation using neural networks. In some implementations, a text in one language is translated into a second language using a neural network model. The model can include an encoder neural network comprising a plurality of bidirectional recurrent neural network layers. The encoding vectors are processed using a multi-headed attention module configured to generate multiple attention context vectors for each encoding vector. A decoder neural network generates a sequence of decoder output vectors using the attention context vectors. The decoder output vectors can represent distributions over various language elements of the second language, allowing a translation of the text into the second language to be determined based on the sequence of decoder output vectors.

公开(公告)号：US20180341860A1

公开(公告)日：2018-11-29

申请号：US16021971

申请日：2018-06-28

Applicant: Google LLC

Inventor： Noam M. Shazeer ,  Aidan Nicholas Gomez ,  Lukasz Mieczyslaw Kaiser ,  Jakob D. Uszkoreit ,  Llion Owen Jones ,  Niki J. Parmar ,  Illia Polosukhin ,  Ashish Teku Vaswani

IPC: G06N3/08 ,  G06N3/04

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output sequence from an input sequence. In one aspect, one of the systems includes an encoder neural network configured to receive the input sequence and generate encoded representations of the network inputs, the encoder neural network comprising a sequence of one or more encoder subnetworks, each encoder subnetwork configured to receive a respective encoder subnetwork input for each of the input positions and to generate a respective subnetwork output for each of the input positions, and each encoder subnetwork comprising: an encoder self-attention sub-layer that is configured to receive the subnetwork input for each of the input positions and, for each particular input position in the input order: apply an attention mechanism over the encoder subnetwork inputs using one or more queries derived from the encoder subnetwork input at the particular input position.

公开(公告)号：US11494561B2

公开(公告)日：2022-11-08

申请号：US16984337

申请日：2020-08-04

Applicant: Google LLC

Inventor： Noam M. Shazeer ,  Aidan Nicholas Gomez ,  Lukasz Mieczyslaw Kaiser ,  Jakob D. Uszkoreit ,  Llion Owen Jones ,  Niki J. Parmar ,  Ashish Teku Vaswani

IPC: G06F40/284 ,  G06K9/62 ,  G06N3/04 ,  G06N3/08

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media for training a machine learning model to perform multiple machine learning tasks from multiple machine learning domains. One system includes a machine learning model that includes multiple input modality neural networks corresponding to respective different modalities and being configured to map received data inputs of the corresponding modality to mapped data inputs from a unified representation space; an encoder neural network configured to process mapped data inputs from the unified representation space to generate respective encoder data outputs; a decoder neural network configured to process encoder data outputs to generate respective decoder data outputs from the unified representation space; and multiple output modality neural networks corresponding to respective different modalities and being configured to map decoder data outputs to data outputs of the corresponding modality.

公开(公告)号：US11138392B2

公开(公告)日：2021-10-05

申请号：US16521780

申请日：2019-07-25

Applicant: Google LLC

Inventor： Zhifeng Chen ,  Macduff Richard Hughes ,  Yonghui Wu ,  Michael Schuster ,  Xu Chen ,  Llion Owen Jones ,  Niki J. Parmar ,  George Foster ,  Orhan Firat ,  Ankur Bapna ,  Wolfgang Macherey ,  Melvin Jose Johnson Premkumar

IPC: G06F40/58 ,  G06N3/08

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for machine translation using neural networks. In some implementations, a text in one language is translated into a second language using a neural network model. The model can include an encoder neural network comprising a plurality of bidirectional recurrent neural network layers. The encoding vectors are processed using a multi-headed attention module configured to generate multiple attention context vectors for each encoding vector. A decoder neural network generates a sequence of decoder output vectors using the attention context vectors. The decoder output vectors can represent distributions over various language elements of the second language, allowing a translation of the text into the second language to be determined based on the sequence of decoder output vectors.

公开(公告)号：US20200034436A1

公开(公告)日：2020-01-30

申请号：US16521780

申请日：2019-07-25

Applicant: Google LLC

Inventor： Zhifeng Chen ,  Macduff Richard Hughes ,  Yonghui Wu ,  Michael Schuster ,  Xu Chen ,  Llion Owen Jones ,  Niki J. Parmar ,  George Foster ,  Orhan Firat ,  Ankur Bapna ,  Wolfgang Macherey ,  Melvin Jose Johnson Premkumar

IPC: G06F17/28 ,  G06N3/08

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for machine translation using neural networks. In some implementations, a text in one language is translated into a second language using a neural network model. The model can include an encoder neural network comprising a plurality of bidirectional recurrent neural network layers. The encoding vectors are processed using a multi-headed attention module configured to generate multiple attention context vectors for each encoding vector. A decoder neural network generates a sequence of decoder output vectors using the attention context vectors. The decoder output vectors can represent distributions over various language elements of the second language, allowing a translation of the text into the second language to be determined based on the sequence of decoder output vectors.

公开(公告)号：US20190392319A1

公开(公告)日：2019-12-26

申请号：US16559392

申请日：2019-09-03

Applicant: Google LLC

Inventor： Noam M. Shazeer ,  Aidan Nicholas Gomez ,  Lukasz Mieczyslaw Kaiser ,  Jakob D. Uszkoreit ,  Llion Owen Jones ,  Niki J. Parmar ,  Illia Polosukhin ,  Ashish Teku Vaswani

IPC: G06N3/08 ,  G06N3/04

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output sequence from an input sequence. In one aspect, one of the systems includes an encoder neural network configured to receive the input sequence and generate encoded representations of the network inputs, the encoder neural network comprising a sequence of one or more encoder subnetworks, each encoder subnetwork configured to receive a respective encoder subnetwork input for each of the input positions and to generate a respective subnetwork output for each of the input positions, and each encoder subnetwork comprising: an encoder self-attention sub-layer that is configured to receive the subnetwork input for each of the input positions and, for each particular input position in the input order: apply an attention mechanism over the encoder subnetwork inputs using one or more queries derived from the encoder subnetwork input at the particular input position.

公开(公告)号：US20200372357A1

公开(公告)日：2020-11-26

申请号：US16932422

申请日：2020-07-17

Applicant: Google LLC

Inventor： Noam M. Shazeer ,  Aidan Nicholas Gomez ,  Lukasz Mieczyslaw Kaiser ,  Jakob D. Uszkoreit ,  Llion Owen Jones ,  Niki J. Parmar ,  Illia Polosukhin ,  Ashish Teku Vaswani

IPC: G06N3/08 ,  G06N3/04

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output sequence from an input sequence. In one aspect, one of the systems includes an encoder neural network configured to receive the input sequence and generate encoded representations of the network inputs, the encoder neural network comprising a sequence of one or more encoder subnetworks, each encoder subnetwork configured to receive a respective encoder subnetwork input for each of the input positions and to generate a respective subnetwork output for each of the input positions, and each encoder subnetwork comprising: an encoder self-attention sub-layer that is configured to receive the subnetwork input for each of the input positions and, for each particular input position in the input order: apply an attention mechanism over the encoder subnetwork inputs using one or more queries derived from the encoder subnetwork input at the particular input position.