公开(公告)号：US20250053815A1

公开(公告)日：2025-02-13

申请号：US18806647

申请日：2024-08-15

Applicant: Google LLC

Inventor： William Bradley Fedus ,  Barret Zoph ,  Noam M. Shazeer

IPC: G06N3/082 ,  G06N3/045

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing a machine learning task on a network input to generate a network output. In one aspect, one of the systems includes a neural network configured to perform the machine learning task, the neural network including one or more switch layers.

公开(公告)号：US20250037711A1

公开(公告)日：2025-01-30

申请号：US18912175

申请日：2024-10-10

Applicant: GOOGLE LLC

Inventor： Martin Baeuml ,  Thushan Amarasiriwardena ,  Roberto Pieraccini ,  Vikram Sridar ,  Daniel De Freitas Adiwardana ,  Noam M. Shazeer ,  Quoc Le

IPC: G10L15/183 ,  G06F16/9032 ,  G10L15/22

Abstract: As part of a dialog session between a user and an automated assistant, implementations can receive a stream of audio data that captures a spoken utterance including an assistant query, determine, based on processing the stream of audio data, a set of assistant outputs that are each predicted to be responsive to the assistant query, process, using large language model (LLM) output(s), the assistant outputs and context of the dialog session to generate a set of modified assistant outputs, and cause given modified assistant output, from among the set of modified assistant outputs, to be provided for presentation to the user in response to the spoken utterance. In some implementations, the LLM output(s) can be generated in an offline manner for subsequent use in an online manner. In additional or alternative implementations, the LLM output(s) can be generated in an online manner when the spoken utterance is received.

公开(公告)号：US12148421B2

公开(公告)日：2024-11-19

申请号：US17532794

申请日：2021-11-22

Applicant: GOOGLE LLC

Inventor： Martin Baeuml ,  Thushan Amarasiriwardena ,  Roberto Pieraccini ,  Vikram Sridar ,  Daniel De Freitas Adiwardana ,  Noam M. Shazeer ,  Quoc Le

IPC: G10L15/22 ,  G06F16/9032 ,  G10L15/183

Abstract: As part of a dialog session between a user and an automated assistant, implementations can receive a stream of audio data that captures a spoken utterance including an assistant query, determine, based on processing the stream of audio data, a set of assistant outputs that are each predicted to be responsive to the assistant query, process, using large language model (LLM) output(s), the assistant outputs and context of the dialog session to generate a set of modified assistant outputs, and cause given modified assistant output, from among the set of modified assistant outputs, to be provided for presentation to the user in response to the spoken utterance. In some implementations, the LLM output(s) can be generated in an offline manner for subsequent use in an online manner. In additional or alternative implementations, the LLM output(s) can be generated in an online manner when the spoken utterance is received.

公开(公告)号：US12067476B2

公开(公告)日：2024-08-20

申请号：US18244171

申请日：2023-09-08

Applicant: Google LLC

Inventor： Noam M. Shazeer ,  Azalia Mirhoseini ,  Krzysztof Stanislaw Maziarz

IPC: G06N3/045 ,  G06N3/08

CPC classification number: G06N3/045 ,  G06N3/08

Abstract: A system includes a neural network that includes a Mixture of Experts (MoE) subnetwork between a first neural network layer and a second neural network layer. The MoE subnetwork includes multiple expert neural networks. Each expert neural network is configured to process a first layer output generated by the first neural network layer to generate a respective expert output. The MoE subnetwork further includes a gating subsystem that selects, based on the first layer output, one or more of the expert neural networks and determine a respective weight for each selected expert neural network, provides the first layer output as input to each of the selected expert neural networks, combines the expert outputs generated by the selected expert neural networks in accordance with the weights for the selected expert neural networks to generate an MoE output, and provides the MoE output as input to the second neural network layer.

公开(公告)号：US20240256859A1

公开(公告)日：2024-08-01

申请号：US18403966

申请日：2024-01-04

Applicant: Google LLC

Inventor： Noam M. Shazeer ,  Lukasz Mieczyslaw Kaiser ,  Etienne Pot ,  Mohammad Saleh ,  Ben David Goodrich ,  Peter J. Liu ,  Ryan Sepassi

IPC: G06N3/08 ,  G06N3/045

CPC classification number: G06N3/08 ,  G06N3/045

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output sequence from an input sequence. One of the methods includes, at each of a plurality of generation time steps: generating a combined sequence for the generation time step that includes the input sequence followed by the output tokens that have already been generated as of the generation time step; processing the combined sequence using a self-attention decoder neural network to generate a time step output that defines a score distribution over a set of possible output tokens; and selecting, using the time step output, an output token from the set of possible output tokens as the next output token in the output sequence.

公开(公告)号：US20240144006A1

公开(公告)日：2024-05-02

申请号：US18407299

申请日：2024-01-08

Applicant: Google LLC

Inventor： Noam M. Shazeer ,  Aidan Nicholas Gomez ,  Lukasz Mieczyslaw Kaiser ,  Jakob D. Uszkoreit ,  Llion Owen Jones ,  Niki J. Parmar ,  Illia Polosukhin ,  Ashish Teku Vaswani

IPC: G06N3/08 ,  G06N3/04 ,  G06N3/045 ,  G06N20/00

CPC classification number: G06N3/08 ,  G06N3/04 ,  G06N3/045 ,  G06N20/00

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output sequence from an input sequence. In one aspect, one of the systems includes an encoder neural network configured to receive the input sequence and generate encoded representations of the network inputs, the encoder neural network comprising a sequence of one or more encoder subnetworks, each encoder subnetwork configured to receive a respective encoder subnetwork input for each of the input positions and to generate a respective subnetwork output for each of the input positions, and each encoder subnetwork comprising: an encoder self-attention sub-layer that is configured to receive the subnetwork input for each of the input positions and, for each particular input position in the input order: apply an attention mechanism over the encoder subnetwork inputs using one or more queries derived from the encoder subnetwork input at the particular input position.

公开(公告)号：US11886998B2

公开(公告)日：2024-01-30

申请号：US18096946

申请日：2023-01-13

Applicant: Google LLC

Inventor： Noam M. Shazeer ,  Lukasz Mieczyslaw Kaiser ,  Etienne Pot ,  Mohammad Saleh ,  Ben Goodrich ,  Peter J. Liu ,  Ryan Sepassi

IPC: G06N3/08 ,  G06N3/045

CPC classification number: G06N3/08 ,  G06N3/045

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output sequence from an input sequence. One of the methods includes, at each of a plurality of generation time steps: generating a combined sequence for the generation time step that includes the input sequence followed by the output tokens that have already been generated as of the generation time step; processing the combined sequence using a self-attention decoder neural network to generate a time step output that defines a score distribution over a set of possible output tokens; and selecting, using the time step output, an output token from the set of possible output tokens as the next output token in the output sequence.

公开(公告)号：US11816884B2

公开(公告)日：2023-11-14

申请号：US17867242

申请日：2022-07-18

Applicant: Google LLC

Inventor： Noam M. Shazeer ,  Lukasz Mieczyslaw Kaiser ,  Jakob D. Uszkoreit ,  Niki J. Parmar ,  Ashish Teku Vaswani

IPC: G06V10/82 ,  G06N3/084 ,  G06N3/04 ,  G06T3/40 ,  G06F18/28 ,  G06F18/213 ,  G06F18/21 ,  G06V10/77 ,  G06V10/56

CPC classification number: G06V10/82 ,  G06F18/213 ,  G06F18/217 ,  G06F18/28 ,  G06N3/04 ,  G06N3/084 ,  G06T3/4053 ,  G06V10/56 ,  G06V10/7715

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output image. In one aspect, one of the methods includes generating the output image intensity value by intensity value according to a generation order of pixel—color channel pairs from the output image, comprising, for each particular generation order position in the generation order: generating a current output image representation of a current output image, processing the current output image representation using a decoder neural network to generate a probability distribution over possible intensity values for the pixel—color channel pair at the particular generation order position, wherein the decoder neural network includes one or more local masked self-attention sub-layers; and selecting an intensity value for the pixel—color channel pair at the particular generation order position using the probability distribution.

公开(公告)号：US20230029590A1

公开(公告)日：2023-02-02

申请号：US17876451

申请日：2022-07-28

Applicant: Google LLC

Inventor： Daniel De Freitas Adiwardana ,  Noam M. Shazeer

IPC: G06N3/08 ,  G06K9/62

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for evaluating candidate output sequences using language model neural networks. In particular, an auto-regressive language model neural network is used to generate a candidate output sequence. The same auto-regressive language model neural network is used to evaluate the candidate output sequence to determine rating scores for each of one or more criteria. The rating score(s) are then used to determine whether to provide the candidate output sequence.

公开(公告)号：US20220383119A1

公开(公告)日：2022-12-01

申请号：US17827362

申请日：2022-05-27

Applicant: Google LLC

Inventor： David Richard So ,  Quoc V. Le, Jr. ,  Hanxiao Liu ,  Wojciech Andrzej Manke ,  Zihang Dai ,  Noam M. Shazeer

IPC: G06N3/08

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for performing a machine learning task on a network input to generate a network output. One of the systems includes an attention neural network configured to perform the machine learning task. The attention neural network includes one or more attentions layers that each include a squared ReLU activation layer, a depth-wise convolution layer, or both.