公开(公告)号：US11164066B1

公开(公告)日：2021-11-02

申请号：US15716330

申请日：2017-09-26

Applicant: Google LLC

Inventor： Andrew M. Dai ,  Quoc V. Le ,  David Ha

IPC: G06N3/04 ,  G06N3/08

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing inputs using recurrent neural networks. One of the systems includes a main recurrent neural network comprising one or more recurrent neural network layers and a respective hyper recurrent neural network corresponding to each of the one or more recurrent neural network layers, wherein each hyper recurrent neural network is configured to, at each of a plurality of time steps: process the layer input at the time step to the corresponding recurrent neural network layer, the current layer hidden state of the corresponding recurrent neural network layer, and a current hypernetwork hidden state of the hyper recurrent neural network to generate an updated hypernetwork hidden state.

公开(公告)号：US10528866B1

公开(公告)日：2020-01-07

申请号：US15257539

申请日：2016-09-06

Applicant: Google LLC

Inventor： Andrew M. Dai ,  Quoc V. Le

IPC: G06N3/04 ,  G06N3/08

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a document classification neural network. One of the methods includes training an autoencoder neural network to autoencode input documents, wherein the autoencoder neural network comprises the one or more LSTM neural network layers and an autoencoder output layer, and wherein training the autoencoder neural network comprises determining pre-trained values of the parameters of the one or more LSTM neural network layers from initial values of the parameters of the one or more LSTM neural network layers; and training the document classification neural network on a plurality of training documents to determine trained values of the parameters of the one or more LSTM neural network layers from the pre-trained values of the parameters of the one or more LSTM neural network layers.

公开(公告)号：US20250131251A1

公开(公告)日：2025-04-24

申请号：US18834070

申请日：2023-01-30

Applicant: Google LLC

Inventor： Hanxiao Liu ,  Quoc V. Le ,  Yanqi Zhou ,  Tao Lei ,  Yuzhe Zhao ,  Yanping Huang ,  Nan Du ,  Zhifeng Chen ,  Andrew M. Dai ,  James Laudon

IPC: G06N3/048

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing a machine learning task on a network input to generate a network output. In one aspect, one of the systems includes a neural network configured to perform the machine learning task, the neural network including one or more expert neural network blocks that each include router that performs expert-choice routing between multiple expert neural networks.

公开(公告)号：US20250118401A1

公开(公告)日：2025-04-10

申请号：US17143083

申请日：2021-01-06

Applicant: Google LLC

Inventor： Edward Choi ,  Andrew M. Dai ,  Gerardo Flores ,  Yuan Xue ,  Michael Ward Dusenberry ,  Zhen Xu ,  Yujia Li

IPC: G16H15/00 ,  G06N3/08 ,  G16H50/20

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing data about a medical encounter using neural networks. One of the methods includes obtaining features for a medical encounter associated with the patient, each feature representing a corresponding health event associated with the medical encounter and each of the plurality of features belonging to a vocabulary of possible features that each represent a different health event; and generating respective final embeddings for each of the features for the medical encounter by applying a sequence of one or more self-attention blocks to the features for the medical encounter, wherein each of the one or more self-attention blocks receives a respective block input for each of the features and applies self-attention over the block inputs to generate a respective block output for each of the features.

公开(公告)号：US20240378441A1

公开(公告)日：2024-11-14

申请号：US18661447

申请日：2024-05-10

Applicant: Google LLC

Inventor： Slav Petrov ,  Yonghui Wu ,  Andrew M. Dai ,  David Richard So ,  Dmitry Lepikhin ,  Erica Ann Moreira ,  Gaurav Mishra ,  Jonathan Hudson Clark ,  Maxim Krikun ,  Melvin Jose Johnson Premkumar ,  Nan Du ,  Orhan Firat ,  Rohan Anil ,  Siamak Shakeri ,  Xavier Garcia ,  Yanping Huang ,  Yong Cheng ,  Yuanzhong Xu ,  Yujing Zhang ,  Zachary Alexander Nado ,  Eric Jun Jie Ni ,  Kefan Xiao ,  Vladimir Feinberg ,  Jin Young Sohn ,  Aurko Roy

IPC: G06N3/08

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a neural network to perform any one or more of a variety of machine learning tasks. For example, the neural network can be configured as a generative neural network, e.g., an autoregressive generative neural network.

公开(公告)号：US11868888B1

公开(公告)日：2024-01-09

申请号：US17549746

申请日：2021-12-13

Applicant: Google LLC

Inventor： Andrew M. Dai ,  Quoc V. Le

IPC: G06N3/04 ,  G06N3/08 ,  G06N3/044 ,  G06N3/045

CPC classification number: G06N3/08 ,  G06N3/044 ,  G06N3/045

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a document classification neural network. One of the methods includes training an autoencoder neural network to autoencode input documents, wherein the autoencoder neural network comprises the one or more LSTM neural network layers and an autoencoder output layer, and wherein training the autoencoder neural network comprises determining pre-trained values of the parameters of the one or more LSTM neural network layers from initial values of the parameters of the one or more LSTM neural network layers; and training the document classification neural network on a plurality of training documents to determine trained values of the parameters of the one or more LSTM neural network layers from the pre-trained values of the parameters of the one or more LSTM neural network layers.

公开(公告)号：US20230274151A1

公开(公告)日：2023-08-31

申请号：US17915796

申请日：2021-03-30

Applicant: Google LLC

Inventor： Zhen Xu ,  David Richard So ,  Andrew M. Dai

IPC: G06N3/086 ,  G06N3/045

CPC classification number: G06N3/086 ,  G06N3/045

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for searching for an architecture for a neural network that performs a multi-modal task that requires operating on inputs that each include data from multiple different modalities.

公开(公告)号：US11742087B2

公开(公告)日：2023-08-29

申请号：US16990172

申请日：2020-08-11

Applicant: Google LLC

Inventor： Jonas Beachey Kemp ,  Andrew M. Dai ,  Alvin Rishi Rajkomar

IPC: G16H50/20 ,  G06N3/049 ,  G16H10/60 ,  G16H50/30

CPC classification number: G16H50/20 ,  G06N3/049 ,  G16H10/60 ,  G16H50/30

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for predicting future patient health using neural networks. One of the methods includes receiving electronic health record data for a patient; generating a respective observation embedding for each of the observations, comprising, for each clinical note: processing the sequence of tokens in the clinical note using a clinical note embedding LSTM to generate a respective token embedding for each of the tokens; and generating the observation embedding for the clinical note from the token embeddings; generating an embedded representation, comprising, for each time window: combining the observation embeddings of observations occurring during the time window to generate a patient record embedding; and processing the embedded representation of the electronic health record data using a prediction recurrent neural network to generate a neural network output that characterizes a future health status of the patient.

公开(公告)号：US20210125721A1

公开(公告)日：2021-04-29

申请号：US16990172

申请日：2020-08-11

Applicant: Google LLC

Inventor： Jonas Beachey Kemp ,  Andrew M. Dai ,  Alvin Rishi Rajkomar

IPC: G16H50/20 ,  G06N3/04 ,  G16H10/60 ,  G16H50/30

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for predicting future patient health using neural networks. One of the methods includes receiving electronic health record data for a patient; generating a respective observation embedding for each of the observations, comprising, for each clinical note: processing the sequence of tokens in the clinical note using a clinical note embedding LSTM to generate a respective token embedding for each of the tokens; and generating the observation embedding for the clinical note from the token embeddings; generating an embedded representation, comprising, for each time window: combining the observation embeddings of observations occurring during the time window to generate a patient record embedding; and processing the embedded representation of the electronic health record data using a prediction recurrent neural network to generate a neural network output that characterizes a future health status of the patient.

公开(公告)号：US10803380B2

公开(公告)日：2020-10-13

申请号：US15262959

申请日：2016-09-12

Applicant: Google LLC

Inventor： Andrew M. Dai ,  Quoc V. Le ,  Gregory Sean Corrado

IPC: G06N3/08 ,  G06F40/279

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating document vector representations. One of the methods includes obtaining a new document; selecting a plurality of new document word sets; and determining a vector representation for the new document using a trained neural network system, wherein the trained neural network system comprises: a document embedding layer and a classifier, and wherein determining the vector representation for the new document using the trained neural network system comprises iteratively providing each of the plurality of new document word sets to the trained neural network system to determine the vector representation for the new document using gradient descent.