公开(公告)号：US11188544B1

公开(公告)日：2021-11-30

申请号：US16299130

申请日：2019-03-11

Applicant: Google LLC

Inventor： Hyung-Jin Kim ,  Simon Tong ,  Noam M. Shazeer ,  Michelangelo Diligenti

IPC: G06F16/9535 ,  G06F16/2457 ,  G06F16/951 ,  G06F16/338 ,  G06Q30/02

Abstract: The present disclosure includes systems and techniques relating to ranking search results of a search query. In general, the subject matter described in this specification can be embodied in a computer-implemented method that includes determining a measure of relevance for a document result within a context of a search query for which the document result is returned, the determining being based on a first number in relation to a second number, the first number corresponding to longer views of the document result, and the second number corresponding to at least shorter views of the document result; and outputting the measure of relevance to a ranking engine for ranking of search results, including the document result, for a new search corresponding to the search query. The subject matter described in this specification can also be embodied in various corresponding computer program products, apparatus and systems.

公开(公告)号：US20210019626A1

公开(公告)日：2021-01-21

申请号：US17063034

申请日：2020-10-05

Applicant: Google LLC

Inventor： Noam M. Shazeer

IPC: G06N3/08 ,  G06N3/04 ,  G06F16/901 ,  G06K9/62

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for distributing tensor computations across computing devices. One of the methods includes: receiving specification data that specifies a distribution of tensor computations among a plurality of computing devices, wherein each tensor computation (i) is defined to receive, as input, one or more respective input tensors each having one or more respective input dimensions, (ii) is defined to generate, as output, one or more respective output tensors each having one or more respective output dimensions, or both, wherein the specification data specifies a respective layout for each input and output tensor that assigns each dimension of the input or output tensor to one or more of the plurality of computing devices; assigning, based on the layouts for the input and output tensors, respective device-local operations to each of the computing devices; and causing the tensor computations to be executed.

公开(公告)号：US10796225B2

公开(公告)日：2020-10-06

申请号：US16532381

申请日：2019-08-05

Applicant: Google LLC

Inventor： Noam M. Shazeer

IPC: G06F15/18 ,  G06N3/08 ,  G06N3/04 ,  G06F16/901 ,  G06K9/62 ,  G06F17/11

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for distributing tensor computations across computing devices. One of the methods includes: receiving specification data that specifies a distribution of tensor computations among a plurality of computing devices, wherein each tensor computation (i) is defined to receive, as input, one or more respective input tensors each having one or more respective input dimensions, (ii) is defined to generate, as output, one or more respective output tensors each having one or more respective output dimensions, or both, wherein the specification data specifies a respective layout for each input and output tensor that assigns each dimension of the input or output tensor to one or more of the plurality of computing devices; assigning, based on the layouts for the input and output tensors, respective device-local operations to each of the computing devices; and causing the tensor computations to be executed.

公开(公告)号：US20200279150A1

公开(公告)日：2020-09-03

申请号：US16879187

申请日：2020-05-20

Applicant: Google LLC

Inventor： Noam M. Shazeer ,  Azalia Mirhoseini ,  Krzysztof Stanislaw Maziarz

IPC: G06N3/04 ,  G06N3/08

Abstract: A system includes a neural network that includes a Mixture of Experts (MoE) subnetwork between a first neural network layer and a second neural network layer. The MoE subnetwork includes multiple expert neural networks. Each expert neural network is configured to process a first layer output generated by the first neural network layer to generate a respective expert output. The MoE subnetwork further includes a gating subsystem that selects, based on the first layer output, one or more of the expert neural networks and determine a respective weight for each selected expert neural network, provides the first layer output as input to each of the selected expert neural networks, combines the expert outputs generated by the selected expert neural networks in accordance with the weights for the selected expert neural networks to generate an MoE output, and provides the MoE output as input to the second neural network layer.

公开(公告)号：US10719761B2

公开(公告)日：2020-07-21

申请号：US16393063

申请日：2019-04-24

Applicant: Google LLC

Inventor： Noam M. Shazeer ,  Azalia Mirhoseini ,  Krzysztof Stanislaw Maziarz

IPC: G06N3/04 ,  G06N3/08

Abstract: A system includes a neural network that includes a Mixture of Experts (MoE) subnetwork between a first neural network layer and a second neural network layer. The MoE subnetwork includes multiple expert neural networks. Each expert neural network is configured to process a first layer output generated by the first neural network layer to generate a respective expert output. The MoE subnetwork further includes a gating subsystem that selects, based on the first layer output, one or more of the expert neural networks and determine a respective weight for each selected expert neural network, provides the first layer output as input to each of the selected expert neural networks, combines the expert outputs generated by the selected expert neural networks in accordance with the weights for the selected expert neural networks to generate an MoE output, and provides the MoE output as input to the second neural network layer.

公开(公告)号：US10685012B2

公开(公告)日：2020-06-16

申请号：US15424671

申请日：2017-02-03

Applicant: Google LLC

Inventor： Noam M. Shazeer ,  Colin Hearne Evans ,  Christopher Robert Waterson ,  Ryan P. Doherty

IPC: G06F16/23 ,  G06F16/31 ,  G06F16/332 ,  G06F16/33 ,  G06K9/62 ,  G06F40/44 ,  G06F17/16 ,  G06N5/04

Abstract: Methods, and systems, including computer programs encoded on computer storage media for generating compressed representations from a co-occurrence matrix. A method includes obtaining a set of sub matrices of a co-occurrence matrix, where each row of the co-occurrence matrix corresponds to a feature from a first feature vocabulary and each column of the co-occurrence matrix corresponds to a feature from a second feature vocabulary; selecting a sub matrix, wherein the sub matrix is associated with a particular row block and column block of the co-occurrence matrix; assigning respective d-dimensional initial row and column embedding vectors to each row and column from the particular row and column blocks, respectively; and determining a final row embedding vector and a final column embedding vector by iteratively adjusting the initial row embedding vectors and the initial column embedding vectors using the co-occurrence matrix.

公开(公告)号：US10540962B1

公开(公告)日：2020-01-21

申请号：US15970662

申请日：2018-05-03

Applicant: Google LLC

Inventor： William Chan ,  Navdeep Jaitly ,  Quoc V. Le ,  Oriol Vinyals ,  Noam M. Shazeer

IPC: G10L15/16 ,  G10L15/26 ,  G06F17/22 ,  G10L15/183 ,  G10L25/30

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media for speech recognition. One method includes obtaining an input acoustic sequence, the input acoustic sequence representing an utterance, and the input acoustic sequence comprising a respective acoustic feature representation at each of a first number of time steps; processing the input acoustic sequence using a first neural network to convert the input acoustic sequence into an alternative representation for the input acoustic sequence; processing the alternative representation for the input acoustic sequence using an attention-based Recurrent Neural Network (RNN) to generate, for each position in an output sequence order, a set of substring scores that includes a respective substring score for each substring in a set of substrings; and generating a sequence of substrings that represent a transcription of the utterance.

公开(公告)号：US20190392319A1

公开(公告)日：2019-12-26

申请号：US16559392

申请日：2019-09-03

Applicant: Google LLC

Inventor： Noam M. Shazeer ,  Aidan Nicholas Gomez ,  Lukasz Mieczyslaw Kaiser ,  Jakob D. Uszkoreit ,  Llion Owen Jones ,  Niki J. Parmar ,  Illia Polosukhin ,  Ashish Teku Vaswani

IPC: G06N3/08 ,  G06N3/04

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output sequence from an input sequence. In one aspect, one of the systems includes an encoder neural network configured to receive the input sequence and generate encoded representations of the network inputs, the encoder neural network comprising a sequence of one or more encoder subnetworks, each encoder subnetwork configured to receive a respective encoder subnetwork input for each of the input positions and to generate a respective subnetwork output for each of the input positions, and each encoder subnetwork comprising: an encoder self-attention sub-layer that is configured to receive the subnetwork input for each of the input positions and, for each particular input position in the input order: apply an attention mechanism over the encoder subnetwork inputs using one or more queries derived from the encoder subnetwork input at the particular input position.

公开(公告)号：US20190354812A1

公开(公告)日：2019-11-21

申请号：US16417190

申请日：2019-05-20

Applicant: Google LLC

Inventor： Noam M. Shazeer ,  Jakob D. Uszkoreit ,  Mitchell Thomas Stern

IPC: G06K9/62 ,  G06N7/00 ,  G06N20/00

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for performing parallel generation of output from an autoregressive sequence to sequence model. In one aspect, a blockwise parallel decoding method takes advantage of the fact that some architectures can score sequences in sublinear time. By generating predictions for multiple time steps at once then backing off to a longest prefix validated by the scoring model, the methods can substantially improve the speed of greedy decoding without compromising performance.

公开(公告)号：US20250118064A1

公开(公告)日：2025-04-10

申请号：US18913134

申请日：2024-10-11

Applicant: Google LLC

Inventor： Noam M. Shazeer ,  Lukasz Mieczyslaw Kaiser ,  Jakob D. Uszkoreit ,  Niki J. Parmar ,  Ashish Teku Vaswani

IPC: G06V10/82 ,  G06F18/21 ,  G06F18/213 ,  G06F18/28 ,  G06N3/04 ,  G06N3/084 ,  G06T3/4053 ,  G06V10/56 ,  G06V10/77

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output image. In one aspect, one of the methods includes generating the output image intensity value by intensity value according to a generation order of pixel-color channel pairs from the output image, comprising, for each particular generation order position in the generation order: generating a current output image representation of a current output image, processing the current output image representation using a decoder neural network to generate a probability distribution over possible intensity values for the pixel—color channel pair at the particular generation order position, wherein the decoder neural network includes one or more local masked self-attention sub-layers; and selecting an intensity value for the pixel—color channel pair at the particular generation order position using the probability distribution.