公开(公告)号：US11556786B2

公开(公告)日：2023-01-17

申请号：US16759690

申请日：2018-10-29

Applicant: GOOGLE LLC

Inventor： Noam M. Shazeer ,  Lukasz Mieczyslaw Kaiser ,  Etienne Pot ,  Mohammad Saleh ,  Ben David Goodrich ,  Peter J. Liu ,  Ryan Sepassi

IPC: G06N3/08 ,  G06N3/04

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output sequence from an input sequence. One of the methods includes, at each of a plurality of generation time steps: generating a combined sequence for the generation time step that includes the input sequence followed by the output tokens that have already been generated as of the generation time step; processing the combined sequence using a self-attention decoder neural network to generate a time step output that defines a score distribution over a set of possible output tokens; and selecting, using the time step output, an output token from the set of possible output tokens as the next output token in the output sequence.

公开(公告)号：US20240220796A1

公开(公告)日：2024-07-04

申请号：US18403992

申请日：2024-01-04

Applicant: Google LLC

Inventor： Noam M. Shazeer ,  Lukasz Mieczyslaw Kaiser ,  Etienne Pot ,  Mohammad Saleh ,  Ben David Goodrich ,  Peter J. Liu ,  Ryan Sepassi

IPC: G06N3/08 ,  G06N3/045

CPC classification number: G06N3/08 ,  G06N3/045

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output sequence from an input sequence. One of the methods includes, at each of a plurality of generation time steps: generating a combined sequence for the generation time step that includes the input sequence followed by the output tokens that have already been generated as of the generation time step; processing the combined sequence using a self-attention decoder neural network to generate a time step output that defines a score distribution over a set of possible output tokens; and selecting, using the time step output, an output token from the set of possible output tokens as the next output token in the output sequence.

公开(公告)号：US20200342316A1

公开(公告)日：2020-10-29

申请号：US16759690

申请日：2018-10-29

Applicant: GOOGLE LLC

Inventor： Noam M. Shazeer ,  Lukasz Mieczyslaw Kaiser ,  Etienne Pot ,  Mohammad Saleh ,  Ben David Goodrich ,  Peter J. Liu ,  Ryan Sepassi

IPC: G06N3/08 ,  G06N3/04

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output sequence from an input sequence. One of the methods includes, at each of a plurality of generation time steps: generating a combined sequence for the generation time step that includes the input sequence followed by the output tokens that have already been generated as of the generation time step; processing the combined sequence using a self-attention decoder neural network to generate a time step output that defines a score distribution over a set of possible output tokens; and selecting, using the time step output, an output token from the set of possible output tokens as the next output token in the output sequence.

公开(公告)号：US20240256859A1

公开(公告)日：2024-08-01

申请号：US18403966

申请日：2024-01-04

Applicant: Google LLC

Inventor： Noam M. Shazeer ,  Lukasz Mieczyslaw Kaiser ,  Etienne Pot ,  Mohammad Saleh ,  Ben David Goodrich ,  Peter J. Liu ,  Ryan Sepassi

IPC: G06N3/08 ,  G06N3/045

CPC classification number: G06N3/08 ,  G06N3/045

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output sequence from an input sequence. One of the methods includes, at each of a plurality of generation time steps: generating a combined sequence for the generation time step that includes the input sequence followed by the output tokens that have already been generated as of the generation time step; processing the combined sequence using a self-attention decoder neural network to generate a time step output that defines a score distribution over a set of possible output tokens; and selecting, using the time step output, an output token from the set of possible output tokens as the next output token in the output sequence.

公开(公告)号：US11886998B2

公开(公告)日：2024-01-30

申请号：US18096946

申请日：2023-01-13

Applicant: Google LLC

Inventor： Noam M. Shazeer ,  Lukasz Mieczyslaw Kaiser ,  Etienne Pot ,  Mohammad Saleh ,  Ben Goodrich ,  Peter J. Liu ,  Ryan Sepassi

IPC: G06N3/08 ,  G06N3/045

CPC classification number: G06N3/08 ,  G06N3/045

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output sequence from an input sequence. One of the methods includes, at each of a plurality of generation time steps: generating a combined sequence for the generation time step that includes the input sequence followed by the output tokens that have already been generated as of the generation time step; processing the combined sequence using a self-attention decoder neural network to generate a time step output that defines a score distribution over a set of possible output tokens; and selecting, using the time step output, an output token from the set of possible output tokens as the next output token in the output sequence.

公开(公告)号：US20240211752A1

公开(公告)日：2024-06-27

申请号：US18404014

申请日：2024-01-04

Applicant: Google LLC

Inventor： Noam M. Shazeer ,  Lukasz Mieczyslaw Kaiser ,  Etienne Pot ,  Mohammad Saleh ,  Ben David Goodrich ,  Peter J. Liu ,  Ryan Sepassi

IPC: G06N3/08 ,  G06N3/045

CPC classification number: G06N3/08 ,  G06N3/045

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output sequence from an input sequence. One of the methods includes, at each of a plurality of generation time steps: generating a combined sequence for the generation time step that includes the input sequence followed by the output tokens that have already been generated as of the generation time step; processing the combined sequence using a self-attention decoder neural network to generate a time step output that defines a score distribution over a set of possible output tokens; and selecting, using the time step output, an output token from the set of possible output tokens as the next output token in the output sequence.

公开(公告)号：US20240211751A1

公开(公告)日：2024-06-27

申请号：US18403939

申请日：2024-01-04

Applicant: Google LLC

Inventor： Noam M. Shazeer ,  Lukasz Mieczyslaw Kaiser ,  Etienne Pot ,  Mohammad Saleh ,  Ben David Goodrich ,  Peter J. Liu ,  Ryan Sepassi

IPC: G06N3/08 ,  G06N3/045

CPC classification number: G06N3/08 ,  G06N3/045

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output sequence from an input sequence. One of the methods includes, at each of a plurality of generation time steps: generating a combined sequence for the generation time step that includes the input sequence followed by the output tokens that have already been generated as of the generation time step; processing the combined sequence using a self-attention decoder neural network to generate a time step output that defines a score distribution over a set of possible output tokens; and selecting, using the time step output, an output token from the set of possible output tokens as the next output token in the output sequence.

公开(公告)号：US20230153613A1

公开(公告)日：2023-05-18

申请号：US18096946

申请日：2023-01-13

Applicant: Google LLC

Inventor： Noam M. Shazeer ,  Lukasz Mieczyslaw Kaiser ,  Etienne Pot ,  Mohammad Saleh ,  Ben Goodrich ,  Peter J. Liu ,  Ryan Sepassi

IPC: G06N3/08 ,  G06N3/045

CPC classification number: G06N3/08 ,  G06N3/045

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output sequence from an input sequence. One of the methods includes, at each of a plurality of generation time steps: generating a combined sequence for the generation time step that includes the input sequence followed by the output tokens that have already been generated as of the generation time step; processing the combined sequence using a self-attention decoder neural network to generate a time step output that defines a score distribution over a set of possible output tokens; and selecting, using the time step output, an output token from the set of possible output tokens as the next output token in the output sequence.