公开(公告)号：US20230075716A1

公开(公告)日：2023-03-09

申请号：US17797872

申请日：2021-02-08

Applicant: Google LLC

Inventor： William Chan ,  Chitwan Saharia ,  Geoffrey E. Hinton ,  Mohammad Norouzi ,  Navdeep Jaitly

IPC: G06F40/47 ,  G06F40/284

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for sequence modeling. One of the methods includes receiving an input sequence having a plurality of input positions; determining a plurality of blocks of consecutive input positions; processing the input sequence using a neural network to generate a latent alignment, comprising, at each of a plurality of input time steps: receiving a partial latent alignment from a previous input time step; selecting an input position in each block, wherein the token at the selected input position of the partial latent alignment in each block is a mask token; and processing the partial latent alignment and the input sequence using the neural network to generate a new latent alignment, wherein the new latent alignment comprises, at the selected input position in each block, an output token or a blank token; and generating, using the latent alignment, an output sequence.

公开(公告)号：US20210390271A1

公开(公告)日：2021-12-16

申请号：US17459111

申请日：2021-08-27

Applicant: Google LLC

Inventor： Mohammad Norouzi ,  Zhifeng Chen ,  Yonghui Wu ,  Michael Schuster ,  Quoc V. Le

IPC: G06F40/58 ,  G06F40/44 ,  G06N3/04

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for neural machine translation. The method comprises obtaining a first sequence of words in a source language, generating a modified sequence of words in the source language by inserting a word boundary symbol only at the beginning of each word in the first sequence of words and not at the end of each word, dividing the modified sequence of words into wordpieces using a wordpiece model, generating, from the wordpieces, an input sequence of input tokens for a neural machine translation system; and generating an output sequence of words using the neural machine translation system based on the input sequence of input tokens.

公开(公告)号：US10769191B2

公开(公告)日：2020-09-08

申请号：US14576907

申请日：2014-12-19

Applicant: Google LLC

Inventor： Gregory Sean Corrado ,  Tomas Mikolov ,  Samy Bengio ,  Yoram Singer ,  Jonathon Shlens ,  Andrea L. Frome ,  Jeffrey Adgate Dean ,  Mohammad Norouzi

IPC: G06F16/30 ,  G06F16/35 ,  G06F16/50

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for classifying data objects. One of the methods includes obtaining data that associates each term in a vocabulary of terms with a respective high-dimensional representation of the term; obtaining classification data for a data object, wherein the classification data includes a respective score for each of a plurality of categories, and wherein each of the categories is associated with a respective category label; computing an aggregate high-dimensional representation for the data object from high-dimensional representations for the category labels associated with the categories and the respective scores; identifying a first term in the vocabulary of terms having a high-dimensional representation that is closest to the aggregate high-dimensional representation; and selecting the first term as a category label for the data object.

公开(公告)号：US20190286984A1

公开(公告)日：2019-09-19

申请号：US16351104

申请日：2019-03-12

Applicant: Google LLC

Inventor： Vijay Vasudevan ,  Mohammad Norouzi ,  George Edward Dahl ,  Manoj Kumar Sivaraj

IPC: G06N3/08 ,  G06N3/04

Abstract: A method of determining a final architecture for a neural network (NN) for performing a particular NN task is described. The method includes: maintaining a sequence of classifiers, wherein each classifier has been trained to process an input candidate architecture and to assign a score label to the input candidate architecture that defines whether the input candidate architecture is accepted or rejected from further consideration; repeatedly performing the following operations: sampling, from a search space, a batch of candidate architectures; for each candidate architecture: determining whether the candidate architecture is accepted by all of the classifiers in the sequence of classifiers; in response to a determination that the candidate architecture is accepted by all classifiers, adding the candidate architecture to a surviving set of candidate architectures; and selecting a candidate architecture from the surviving set as the final architecture for the neural network for performing the particular NN task.

公开(公告)号：US12165289B2

公开(公告)日：2024-12-10

申请号：US18227120

申请日：2023-07-27

Applicant: Google LLC

Inventor： Chitwan Saharia ,  Jonathan Ho ,  William Chan ,  Tim Salimans ,  David Fleet ,  Mohammad Norouzi

IPC: G06T5/70 ,  G06N3/045 ,  G06N3/08 ,  G06T3/4007 ,  G06T5/50

Abstract: A method includes receiving, by a computing device, training data comprising a plurality of pairs of images, wherein each pair comprises an image and at least one corresponding target version of the image. The method also includes training a neural network based on the training data to predict an enhanced version of an input image, wherein the training of the neural network comprises applying a forward Gaussian diffusion process that adds Gaussian noise to the at least one corresponding target version of each of the plurality of pairs of images to enable iterative denoising of the input image, wherein the iterative denoising is based on a reverse Markov chain associated with the forward Gaussian diffusion process. The method additionally includes outputting the trained neural network.

公开(公告)号：US20240249456A1

公开(公告)日：2024-07-25

申请号：US18624960

申请日：2024-04-02

Applicant: Google LLC

Inventor： Chitwan Saharia ,  William Chan ,  Mohammad Norouzi ,  Saurabh Saxena ,  Yi Li ,  Jay Ha Whang ,  David James Fleet ,  Jonathan Ho

IPC: G06T11/60 ,  G06F40/284 ,  G06F40/40 ,  G06N3/08 ,  G06T3/4053 ,  G06T5/70

CPC classification number: G06T11/60 ,  G06F40/284 ,  G06F40/40 ,  G06N3/08 ,  G06T3/4053 ,  G06T5/70

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating images. In one aspect, a method includes: receiving an input text prompt including a sequence of text tokens in a natural language; processing the input text prompt using a text encoder neural network to generate a set of contextual embeddings of the input text prompt; and processing the contextual embeddings through a sequence of generative neural networks to generate a final output image that depicts a scene that is described by the input text prompt.

公开(公告)号：US20240062062A1

公开(公告)日：2024-02-22

申请号：US18376362

申请日：2023-10-03

Applicant: Google LLC

Inventor： Samuel Bengio ,  Mohammad Norouzi ,  Benoit Steiner ,  Jeffrey Adgate Dean ,  Hieu Hy Pham ,  Azalia Mirhoseini ,  Quoc V. Le ,  Naveen Kumar ,  Yuefeng Zhou ,  Rasmus Munk Larsen

IPC: G06N3/08 ,  G06N5/04 ,  G06N3/10 ,  G06N3/044 ,  G06N3/045

CPC classification number: G06N3/08 ,  G06N5/04 ,  G06N3/105 ,  G06N3/044 ,  G06N3/045

Abstract: A method for determining a placement for machine learning model operations across multiple hardware devices is described. The method includes receiving data specifying a machine learning model to be placed for distributed processing on multiple hardware devices; generating, from the data, a sequence of operation embeddings, each operation embedding in the sequence characterizing respective operations necessary to perform the processing of the machine learning model; processing the sequence of operation embeddings using a placement recurrent neural network in accordance with first values of a plurality network parameters of the placement recurrent neural network to generate a network output that defines a placement of the operations characterized by the operation embeddings in the sequence across the plurality of devices; and scheduling the machine learning model for processing by the multiple hardware devices by placing the operations on the multiple devices according to the placement defined by the network output.

公开(公告)号：US20230342616A1

公开(公告)日：2023-10-26

申请号：US18343579

申请日：2023-06-28

Applicant: Google LLC

Inventor： Ting Chen ,  Simon Komblith ,  Mohammad Norouzi ,  Geoffrey Everest Hinton ,  Kevin Jordan Swersky

IPC: G06N3/084 ,  G06F18/241 ,  G06F18/214 ,  G06V10/774 ,  G06V10/764 ,  G06V10/778 ,  G06N3/08 ,  G06F18/21

CPC classification number: G06N3/084 ,  G06F18/2155 ,  G06F18/2178 ,  G06F18/241 ,  G06N3/08 ,  G06V10/764 ,  G06V10/7753 ,  G06V10/7788 ,  G06T2207/20081

Abstract: Systems, methods, and computer program products for performing semi-supervised contrastive learning of visual representations are provided. For example, the present disclosure provides systems and methods that leverage particular data augmentation schemes and a learnable nonlinear transformation between the representation and the contrastive loss to provide improved visual representations. Further, the present disclosure also provides improvements for semi-supervised contrastive learning. For example, computer-implemented method may include performing semi-supervised contrastive learning based on a set of one or more unlabeled training data, generating an image classification model based on a portion of a plurality of layers in a projection head neural network used in performing the contrastive learning, performing fine-tuning of the image classification model based on a set of one or more labeled training data, and after performing the fine-tuning, distilling the image classification model to a student model comprising a relatively smaller number of parameters than the image classification model.

公开(公告)号：US11797839B2

公开(公告)日：2023-10-24

申请号：US16174126

申请日：2018-10-29

Applicant: Google LLC

Inventor： Mohammad Norouzi ,  Daniel Aaron Abolafia ,  Quoc V. Le

IPC: G06N3/08 ,  G06N3/04 ,  G06N3/044

CPC classification number: G06N3/08 ,  G06N3/044

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a neural network using a priority queue. One of the methods includes maintaining data identifying a set of K output sequences that were previously generated; selecting at least one of the output sequences from the set of output sequences; for each selected output sequence, determining a respective score; determining, for each selected sequence, a respective first update to the current values of the controller parameters; generating a batch of new output sequences using the controller neural network; obtaining a respective reward for each of the new output sequences; determining, from the new output sequences and the output sequences in the maintained data, the K output sequences that have the highest rewards; and modifying the maintained data.

公开(公告)号：US20230325658A1

公开(公告)日：2023-10-12

申请号：US18010426

申请日：2021-09-02

Applicant: Google LLC

Inventor： Nanxin Chen ,  Byungha Chun ,  William Chan ,  Ron J. Weiss ,  Mohammad Norouzi ,  Yu Zhang ,  Yonghui Wu

IPC: G06V10/82 ,  G06N3/08 ,  G10L13/02 ,  G10L25/18 ,  G10L25/30 ,  G06V10/764 ,  G06V10/26

CPC classification number: G06N3/08 ,  G06V10/26 ,  G06V10/764 ,  G06V10/82 ,  G10L13/02 ,  G10L25/18 ,  G10L25/30

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating outputs conditioned on network inputs using neural networks. In one aspect, a method comprises obtaining the network input; initializing a current network output; and generating the final network output by updating the current network output at each of a plurality of iterations, wherein each iteration corresponds to a respective noise level, and wherein the updating comprises, at each iteration: processing a model input for the iteration comprising (i) the current network output and (ii) the network input using a noise estimation neural network that is configured to process the model input to generate a noise output, wherein the noise output comprises a respective noise estimate for each value in the current network output; and updating the current network output using the noise estimate and the noise level for the iteration.