公开(公告)号：US12254411B2

公开(公告)日：2025-03-18

申请号：US17175567

申请日：2021-02-12

Applicant: Google LLC

Inventor： Noam M. Shazeer

IPC: G06N3/08 ,  G06N3/048 ,  G06N3/082

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing a machine learning task on a network input to generate a network output. In one aspect, one of the systems includes an attention neural network configured to perform the machine learning task, the attention neural network including one or more attention layers, each attention layer comprising an attention sub-layer and a feed-forward sub-layer that applies an element-wise multiplication between two vectors generated as a result of two different linear transformations performed on the same attended layer input.

公开(公告)号：US20240211752A1

公开(公告)日：2024-06-27

申请号：US18404014

申请日：2024-01-04

Applicant: Google LLC

Inventor： Noam M. Shazeer ,  Lukasz Mieczyslaw Kaiser ,  Etienne Pot ,  Mohammad Saleh ,  Ben David Goodrich ,  Peter J. Liu ,  Ryan Sepassi

IPC: G06N3/08 ,  G06N3/045

CPC classification number: G06N3/08 ,  G06N3/045

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output sequence from an input sequence. One of the methods includes, at each of a plurality of generation time steps: generating a combined sequence for the generation time step that includes the input sequence followed by the output tokens that have already been generated as of the generation time step; processing the combined sequence using a self-attention decoder neural network to generate a time step output that defines a score distribution over a set of possible output tokens; and selecting, using the time step output, an output token from the set of possible output tokens as the next output token in the output sequence.

公开(公告)号：US20240211751A1

公开(公告)日：2024-06-27

申请号：US18403939

申请日：2024-01-04

Applicant: Google LLC

Inventor： Noam M. Shazeer ,  Lukasz Mieczyslaw Kaiser ,  Etienne Pot ,  Mohammad Saleh ,  Ben David Goodrich ,  Peter J. Liu ,  Ryan Sepassi

IPC: G06N3/08 ,  G06N3/045

CPC classification number: G06N3/08 ,  G06N3/045

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output sequence from an input sequence. One of the methods includes, at each of a plurality of generation time steps: generating a combined sequence for the generation time step that includes the input sequence followed by the output tokens that have already been generated as of the generation time step; processing the combined sequence using a self-attention decoder neural network to generate a time step output that defines a score distribution over a set of possible output tokens; and selecting, using the time step output, an output token from the set of possible output tokens as the next output token in the output sequence.

公开(公告)号：US11954594B1

公开(公告)日：2024-04-09

申请号：US17315695

申请日：2021-05-10

Applicant: Google LLC

Inventor： Samy Bengio ,  Oriol Vinyals ,  Navdeep Jaitly ,  Noam M. Shazeer

IPC: G06N3/08

CPC classification number: G06N3/08

Abstract: This document generally describes a neural network training system, including one or more computers, that trains a recurrent neural network (RNN) to receive an input, e.g., an input sequence, and to generate a sequence of outputs from the input sequence. In some implementations, training can include, for each position after an initial position in a training target sequence, selecting a preceding output of the RNN to provide as input to the RNN at the position, including determining whether to select as the preceding output (i) a true output in a preceding position in the output order or (ii) a value derived from an output of the RNN for the preceding position in an output order generated in accordance with current values of the parameters of the recurrent neural network.

公开(公告)号：US20230222318A1

公开(公告)日：2023-07-13

申请号：US18009841

申请日：2021-06-30

Applicant: Google LLC

Inventor： Dmitry Lepikhin ,  Yanping Huang ,  Orhan Firat ,  Maxim Krikun ,  Dehao Chen ,  Noam M. Shazeer ,  HyoukJoong Lee ,  Yuanzhong Xu ,  Zhifeng Chen

IPC: G06N3/042 ,  G06N3/098

CPC classification number: G06N3/042 ,  G06N3/098

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing machine learning task on a network input to generate a network output. In one aspect, one of the systems includes an attention neural network configured to perform the machine learning task, the attention neural network including one or more attention layers, each attention layer comprising an attention sub-layer and a feed-forward sub-layer. Some or all of the attention layers have a feed-forward sub-layer that applies conditional computation to the inputs to the sub-layer.

公开(公告)号：US11681954B2

公开(公告)日：2023-06-20

申请号：US16682611

申请日：2019-11-13

Applicant: Google LLC

Inventor： Noam M. Shazeer ,  Jakob D. Uszkoreit ,  Mitchell Thomas Stern

IPC: G06N20/20 ,  G06N20/00 ,  G06N7/00 ,  G06F18/21 ,  G06F18/22 ,  G06V10/764 ,  G06V10/82

CPC classification number: G06N20/20 ,  G06F18/2185 ,  G06F18/22 ,  G06N7/00 ,  G06N20/00 ,  G06V10/764 ,  G06V10/82

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for performing parallel generation of output from an autoregressive sequence to sequence model. In one aspect, a blockwise parallel decoding method takes advantage of the fact that some architectures can score sequences in sublinear time. By generating predictions for multiple time steps at once then backing off to a longest prefix validated by the scoring model, the methods can substantially improve the speed of greedy decoding without compromising performance.

公开(公告)号：US20230153613A1

公开(公告)日：2023-05-18

申请号：US18096946

申请日：2023-01-13

Applicant: Google LLC

Inventor： Noam M. Shazeer ,  Lukasz Mieczyslaw Kaiser ,  Etienne Pot ,  Mohammad Saleh ,  Ben Goodrich ,  Peter J. Liu ,  Ryan Sepassi

IPC: G06N3/08 ,  G06N3/045

CPC classification number: G06N3/08 ,  G06N3/045

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output sequence from an input sequence. One of the methods includes, at each of a plurality of generation time steps: generating a combined sequence for the generation time step that includes the input sequence followed by the output tokens that have already been generated as of the generation time step; processing the combined sequence using a self-attention decoder neural network to generate a time step output that defines a score distribution over a set of possible output tokens; and selecting, using the time step output, an output token from the set of possible output tokens as the next output token in the output sequence.

公开(公告)号：US20230074406A1

公开(公告)日：2023-03-09

申请号：US17532794

申请日：2021-11-22

Applicant: GOOGLE LLC

Inventor： Martin Baeuml ,  Thushan Amarasiriwardena ,  Roberto Pieraccini ,  Vikram Sridar ,  Daniel De Freitas Adiwardana ,  Noam M. Shazeer ,  Quoc Le

IPC: G10L15/183 ,  G10L15/22

Abstract: As part of a dialog session between a user and an automated assistant, implementations can receive a stream of audio data that captures a spoken utterance including an assistant query, determine, based on processing the stream of audio data, a set of assistant outputs that are each predicted to be responsive to the assistant query, process, using large language model (LLM) output(s), the assistant outputs and context of the dialog session to generate a set of modified assistant outputs, and cause given modified assistant output, from among the set of modified assistant outputs, to be provided for presentation to the user in response to the spoken utterance. In some implementations, the LLM output(s) can be generated in an offline manner for subsequent use in an online manner. In additional or alternative implementations, the LLM output(s) can be generated in an online manner when the spoken utterance is received.

公开(公告)号：US20200372357A1

公开(公告)日：2020-11-26

申请号：US16932422

申请日：2020-07-17

Applicant: Google LLC

Inventor： Noam M. Shazeer ,  Aidan Nicholas Gomez ,  Lukasz Mieczyslaw Kaiser ,  Jakob D. Uszkoreit ,  Llion Owen Jones ,  Niki J. Parmar ,  Illia Polosukhin ,  Ashish Teku Vaswani

IPC: G06N3/08 ,  G06N3/04

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output sequence from an input sequence. In one aspect, one of the systems includes an encoder neural network configured to receive the input sequence and generate encoded representations of the network inputs, the encoder neural network comprising a sequence of one or more encoder subnetworks, each encoder subnetwork configured to receive a respective encoder subnetwork input for each of the input positions and to generate a respective subnetwork output for each of the input positions, and each encoder subnetwork comprising: an encoder self-attention sub-layer that is configured to receive the subnetwork input for each of the input positions and, for each particular input position in the input order: apply an attention mechanism over the encoder subnetwork inputs using one or more queries derived from the encoder subnetwork input at the particular input position.

公开(公告)号：US10789427B2

公开(公告)日：2020-09-29

申请号：US16689025

申请日：2019-11-19

Applicant: Google LLC

Inventor： Noam M. Shazeer ,  Aidan Nicholas Gomez ,  Lukasz Mieczyslaw Kaiser ,  Jakob D. Uszkoreit ,  Llion Owen Jones ,  Niki J. Parmar ,  Ashish Teku Vaswani

IPC: G06F40/284 ,  G06K9/62 ,  G06N3/04 ,  G06N3/08

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media for training a machine learning model to perform multiple machine learning tasks from multiple machine learning domains. One system includes a machine learning model that includes multiple input modality neural networks corresponding to respective different modalities and being configured to map received data inputs of the corresponding modality to mapped data inputs from a unified representation space; an encoder neural network configured to process mapped data inputs from the unified representation space to generate respective encoder data outputs; a decoder neural network configured to process encoder data outputs to generate respective decoder data outputs from the unified representation space; and multiple output modality neural networks corresponding to respective different modalities and being configured to map decoder data outputs to data outputs of the corresponding modality.