公开(公告)号：US10719764B2

公开(公告)日：2020-07-21

申请号：US16559392

申请日：2019-09-03

Applicant: Google LLC

Inventor： Noam M. Shazeer ,  Aidan Nicholas Gomez ,  Lukasz Mieczyslaw Kaiser ,  Jakob D. Uszkoreit ,  Llion Owen Jones ,  Niki J. Parmar ,  Illia Polosukhin ,  Ashish Teku Vaswani

IPC: G06N3/08 ,  G06N3/04

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output sequence from an input sequence. In one aspect, one of the systems includes an encoder neural network configured to receive the input sequence and generate encoded representations of the network inputs, the encoder neural network comprising a sequence of one or more encoder subnetworks, each encoder subnetwork configured to receive a respective encoder subnetwork input for each of the input positions and to generate a respective subnetwork output for each of the input positions, and each encoder subnetwork comprising: an encoder self-attention sub-layer that is configured to receive the subnetwork input for each of the input positions and, for each particular input position in the input order: apply an attention mechanism over the encoder subnetwork inputs using one or more queries derived from the encoder subnetwork input at the particular input position.

公开(公告)号：US20200151760A1

公开(公告)日：2020-05-14

申请号：US16740014

申请日：2020-01-10

Applicant: Google LLC

Inventor： Ross Koningstein ,  Valentin Spitkovsky ,  Georges Harik ,  Noam M. Shazeer

IPC: G06Q30/02 ,  G06F16/35 ,  G06F16/33 ,  G06F16/332 ,  G06F16/68 ,  G06F16/31

Abstract: Keyword suggestions that are category-aware (and field-proven) may be used to help advertisers better target the serving of their ads, and may reduce unused ad spot inventory. The advertiser can enter ad information, such as a creative, a landing Webpage, other keywords, etc. for example. A keyword facility may use this entered ad information as seed information to infer one or more categories. It may then request that the advertiser confirm or deny some basic feedback information (e.g., categories, Webpage information, etc.). For example, an advertiser may be provided with candidate categories and may be asked to confirm (e.g., using checkboxes) which of the categories are relevant to their ad. Keywords may be determined using at least the categories. The determined keywords may be provided to the advertiser as suggested keywords, or may automatically populate ad serving constraint information as targeting keywords. The ad server system can run a trial on the determined keywords to qualify or disqualify them as targeting keyword.

公开(公告)号：US20200089755A1

公开(公告)日：2020-03-19

申请号：US16689025

申请日：2019-11-19

Applicant: Google LLC

Inventor： Noam M. Shazeer ,  Aidan Nicholas Gomez ,  Lukasz Mieczyslaw Kaiser ,  Jakob D. Uszkoreit ,  Llion Owen Jones ,  Niki J. Parmar ,  Ashish Teku Vaswani

IPC: G06F17/27 ,  G06N3/08 ,  G06N3/04 ,  G06K9/62

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media for training a machine learning model to perform multiple machine learning tasks from multiple machine learning domains. One system includes a machine learning model that includes multiple input modality neural networks corresponding to respective different modalities and being configured to map received data inputs of the corresponding modality to mapped data inputs from a unified representation space; an encoder neural network configured to process mapped data inputs from the unified representation space to generate respective encoder data outputs; a decoder neural network configured to process encoder data outputs to generate respective decoder data outputs from the unified representation space; and multiple output modality neural networks corresponding to respective different modalities and being configured to map decoder data outputs to data outputs of the corresponding modality.

公开(公告)号：US10452978B2

公开(公告)日：2019-10-22

申请号：US16021971

申请日：2018-06-28

Applicant: Google LLC

Inventor： Noam M. Shazeer ,  Aidan Nicholas Gomez ,  Lukasz Mieczyslaw Kaiser ,  Jakob D. Uszkoreit ,  Llion Owen Jones ,  Niki J. Parmar ,  Illia Polosukhin ,  Ashish Teku Vaswani

IPC: G06F15/18 ,  G06N3/08 ,  G06N3/04

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output sequence from an input sequence. In one aspect, one of the systems includes an encoder neural network configured to receive the input sequence and generate encoded representations of the network inputs, the encoder neural network comprising a sequence of one or more encoder subnetworks, each encoder subnetwork configured to receive a respective encoder subnetwork input for each of the input positions and to generate a respective subnetwork output for each of the input positions, and each encoder subnetwork comprising: an encoder self-attention sub-layer that is configured to receive the subnetwork input for each of the input positions and, for each particular input position in the input order: apply an attention mechanism over the encoder subnetwork inputs using one or more queries derived from the encoder subnetwork input at the particular input position.

公开(公告)号：US10229166B1

公开(公告)日：2019-03-12

申请号：US15793773

申请日：2017-10-25

Applicant: Google LLC

Inventor： Hyung-Jin Kim ,  Simon Tong ,  Noam M. Shazeer ,  Michelangelo Diligenti

IPC: G06F17/30 ,  G06F7/00

Abstract: The present disclosure includes systems and techniques relating to ranking search results of a search query. In general, the subject matter described in this specification can be embodied in a computer-implemented method that includes determining a measure of relevance for a document result within a context of a search query for which the document result is returned, the determining being based on a first number in relation to a second number, the first number corresponding to longer views of the document result, and the second number corresponding to at least shorter views of the document result; and outputting the measure of relevance to a ranking engine for ranking of search results, including the document result, for a new search corresponding to the search query. The subject matter described in this specification can also be embodied in various corresponding computer program products, apparatus and systems.

公开(公告)号：US20240428071A1

公开(公告)日：2024-12-26

申请号：US18823611

申请日：2024-09-03

Applicant: Google LLC

Inventor： David Richard So ,  Quoc V. Le ,  Hanxiao Liu ,  Wojciech Andrzej Manke ,  Zihang Dai ,  Noam M. Shazeer

IPC: G06N3/08

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for performing a machine learning task on a network input to generate a network output. One of the systems includes an attention neural network configured to perform the machine learning task. The attention neural network includes one or more attentions layers that each include a squared ReLU activation layer, a depth-wise convolution layer, or both.

公开(公告)号：US12100391B2

公开(公告)日：2024-09-24

申请号：US17450235

申请日：2021-10-07

Applicant: Google LLC

Inventor： William Chan ,  Navdeep Jaitly ,  Quoc V. Le ,  Oriol Vinyals ,  Noam M. Shazeer

IPC: G10L15/16 ,  G06F40/12 ,  G06F40/197 ,  G06N3/044 ,  G06N3/045 ,  G10L15/183 ,  G10L15/26 ,  G10L25/30

CPC classification number: G10L15/16 ,  G06F40/12 ,  G06F40/197 ,  G06N3/044 ,  G06N3/045 ,  G10L15/183 ,  G10L15/26 ,  G10L25/30

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media for speech recognition. One method includes obtaining an input acoustic sequence, the input acoustic sequence representing an utterance, and the input acoustic sequence comprising a respective acoustic feature representation at each of a first number of time steps; processing the input acoustic sequence using a first neural network to convert the input acoustic sequence into an alternative representation for the input acoustic sequence; processing the alternative representation for the input acoustic sequence using an attention-based Recurrent Neural Network (RNN) to generate, for each position in an output sequence order, a set of substring scores that includes a respective substring score for each substring in a set of substrings; and generating a sequence of substrings that represent a transcription of the utterance.

公开(公告)号：US20240220796A1

公开(公告)日：2024-07-04

申请号：US18403992

申请日：2024-01-04

Applicant: Google LLC

Inventor： Noam M. Shazeer ,  Lukasz Mieczyslaw Kaiser ,  Etienne Pot ,  Mohammad Saleh ,  Ben David Goodrich ,  Peter J. Liu ,  Ryan Sepassi

IPC: G06N3/08 ,  G06N3/045

CPC classification number: G06N3/08 ,  G06N3/045

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output sequence from an input sequence. One of the methods includes, at each of a plurality of generation time steps: generating a combined sequence for the generation time step that includes the input sequence followed by the output tokens that have already been generated as of the generation time step; processing the combined sequence using a self-attention decoder neural network to generate a time step output that defines a score distribution over a set of possible output tokens; and selecting, using the time step output, an output token from the set of possible output tokens as the next output token in the output sequence.

公开(公告)号：US20230351188A1

公开(公告)日：2023-11-02

申请号：US18349089

申请日：2023-07-07

Applicant: Google LLC

Inventor： William Bradley Fedus ,  Barret Zoph ,  Noam M. Shazeer

IPC: G06N3/082 ,  G06N3/045

CPC classification number: G06N3/082 ,  G06N3/045

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing a machine learning task on a network input to generate a network output. In one aspect, one of the systems includes a neural network configured to perform the machine learning task, the neural network including one or more switch layers.

公开(公告)号：US11494561B2

公开(公告)日：2022-11-08

申请号：US16984337

申请日：2020-08-04

Applicant: Google LLC

Inventor： Noam M. Shazeer ,  Aidan Nicholas Gomez ,  Lukasz Mieczyslaw Kaiser ,  Jakob D. Uszkoreit ,  Llion Owen Jones ,  Niki J. Parmar ,  Ashish Teku Vaswani

IPC: G06F40/284 ,  G06K9/62 ,  G06N3/04 ,  G06N3/08

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media for training a machine learning model to perform multiple machine learning tasks from multiple machine learning domains. One system includes a machine learning model that includes multiple input modality neural networks corresponding to respective different modalities and being configured to map received data inputs of the corresponding modality to mapped data inputs from a unified representation space; an encoder neural network configured to process mapped data inputs from the unified representation space to generate respective encoder data outputs; a decoder neural network configured to process encoder data outputs to generate respective decoder data outputs from the unified representation space; and multiple output modality neural networks corresponding to respective different modalities and being configured to map decoder data outputs to data outputs of the corresponding modality.