公开(公告)号：US20230022151A1

公开(公告)日：2023-01-26

申请号：US17860691

申请日：2022-07-08

Applicant: Google LLC

Inventor： Hanjun Dai ,  Bo Dai ,  Hongyu Ren ,  Dale Eric Schuurmans ,  Zihang Dai ,  Mengjiao Yang

IPC: G06F40/20 ,  G06N20/00

Abstract: The present disclosure is directed to machine learning model architectures which provide full attention capability in each attention head while maintaining low computation and memory complexity. Specifically, according to one aspect of the present disclosure, example attention models provided herein can treat the self-attention mechanism as a conditional expectation over embeddings at each location and approximate the conditional distribution with a structured factorization. Each location can attend to all other locations, either via direct attention, or through indirect attention to group representations, which are again conditional expectations of embeddings from corresponding local regions.

公开(公告)号：US20250045577A1

公开(公告)日：2025-02-06

申请号：US18697304

申请日：2021-10-05

Applicant: Google LLC

Inventor： Bo Dai ,  Hanjun Dai ,  Yuan Xue ,  Zia Syed ,  Dale Eric Schuurmans

IPC: G06N3/08

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for performing stochastic optimization using machine learning. One of the methods includes obtaining data defining a multi-stage stochastic optimization (MSSO) problem instance, the data characterizing an observation distribution, an action space, and a cost function; generating a neural network input characterizing the MSSO problem instance from the data; providing the neural network input as input to a neural network that generates, from the network input, a neural network output characterizing parameters of a value function corresponding to the MSSO problem instance; processing the neural network input using the neural network to generate the neural network output; obtaining a new observation determined according to the observation distribution for the MSSO problem instance; determining, using the value function characterized by the network output, an optimal action to take in response to the new observation; and executing the optimal action.

公开(公告)号：US20240112013A1

公开(公告)日：2024-04-04

申请号：US17951889

申请日：2022-09-23

Applicant: Google LLC

Inventor： Hanjun Dai ,  Bo Dai ,  Mengjiao Yang ,  Yuan Xue ,  Dale Eric Schuurmans

IPC: G06N3/08 ,  G06N3/04

CPC classification number: G06N3/08 ,  G06N3/0472

Abstract: The present disclosure is directed to generative models for datasets constrained by marginal constraints. One method includes receiving a request to generate a target dataset based on a marginal constraint for a source dataset. A first object occurs at a source frequency in the source dataset. The marginal constraint indicates a target frequency for the first object. The source dataset encodes a set of co-occurrence frequencies for a plurality of object pairs. A source generative model is accessed. The source generative model includes a first module and a second module that are trained on the source dataset. The second module is updated based on the marginal constraint. An adapted generative model is generated that includes the first module and the updated second module. The target dataset is generated based on the adapted generative model. The first object occurs at the target frequency in the target dataset. The target dataset encodes the set of co-occurrence frequencies for the plurality of object pairs.

公开(公告)号：US11947503B2

公开(公告)日：2024-04-02

申请号：US17351086

申请日：2021-06-17

Applicant: Google LLC

Inventor： Hanjun Dai ,  Azade Nazi ,  Yujia Li ,  Bo Dai ,  Dale Eric Schuurmans

IPC: G06F16/21 ,  G06F16/22 ,  G06N3/045

CPC classification number: G06F16/212 ,  G06F16/2237 ,  G06F16/2246 ,  G06N3/045

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating data defining a graph. In one aspect, a method comprises: sequentially generating a respective edge set for each node in the graph, wherein for each of a plurality of nodes after a first node, generating the edge set for the node comprises: receiving a context embedding for the node that summarizes a respective edge set for each node that precedes the node; generating, based on the context embedding for the node: (i) a respective edge set for the node, and (ii) a respective embedding of the edge set for the node; generating a context embedding for a next node in the ordering of the nodes using the embedding of the edge set for the node; and adding the set of edges defined by the edge set for the node to the graph.

公开(公告)号：US20230289626A1

公开(公告)日：2023-09-14

申请号：US18183410

申请日：2023-03-14

Applicant: Google LLC

Inventor： Hanjun Dai ,  Dale Eric Schuurmans ,  Xinyun Chen ,  Dengyong Zhou ,  Bo Dai ,  Hongyu Ren

IPC: G06N5/022 ,  G06F16/2453

CPC classification number: G06N5/022 ,  G06F16/2453

Abstract: Provided are computing systems, methods, and platforms for negative sampling in knowledge graphs with improved efficiency. A knowledge graph comprising entities and links between the entities can be obtained. A query computation graph comprising nodes and edges can be generated based on the knowledge graph. The nodes of the query computation graph can include anchor nodes, a root node, and intermediate nodes positioned in paths between the anchor nodes and the root node. A node cut of a query of the query computation graph can be determined and can include at least one node that cuts at least one path between each anchor node and the root node of the query computation graph. Negative samples can be identified by bidirectionally traversing the query computation graph in a first direction from the anchor nodes to the node cut and in a second direction from the root node to the node cut.

公开(公告)号：US20240386529A1

公开(公告)日：2024-11-21

申请号：US18667132

申请日：2024-05-17

Applicant: Google LLC

Inventor： Mengjiao Yang ,  Yilun Du ,  Bo Dai ,  Dale Eric Schuurmans

IPC: G06T5/60 ,  G06T5/70 ,  G06T13/00

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output video conditioned on an input. The video generation method can be implemented by a system including one or more computers. The system receives a conditioning input, and initializes a current intermediate representation of the output video. At each of a plurality of iterations, the system updates the current intermediate representation using a first denoising diffusion model and a second denoising diffusion model conditioned on the conditioning input.

公开(公告)号：US20220414067A1

公开(公告)日：2022-12-29

申请号：US17351086

申请日：2021-06-17

Applicant: Google LLC

Inventor： Hanjun Dai ,  Azade Nazi ,  Yujia Li ,  Bo Dai ,  Dale Eric Schuurmans

IPC: G06F16/21 ,  G06N3/04 ,  G06F16/22

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating data defining a graph. In one aspect, a method comprises: sequentially generating a respective edge set for each node in the graph, wherein for each of a plurality of nodes after a first node, generating the edge set for the node comprises: receiving a context embedding for the node that summarizes a respective edge set for each node that precedes the node; generating, based on the context embedding for the node: (i) a respective edge set for the node, and (ii) a respective embedding of the edge set for the node; generating a context embedding for a next node in the ordering of the nodes using the embedding of the edge set for the node; and adding the set of edges defined by the edge set for the node to the graph.

公开(公告)号：US20190114343A1

公开(公告)日：2019-04-18

申请号：US15783685

申请日：2017-10-13

Applicant: Google LLC

Inventor： Ruiqi Guo ,  Bo Dai ,  Sanjiv Kumar

IPC: G06F17/30 ,  G06F15/18

Abstract: The present disclosure provides systems and methods that perform stochastic generative hashing. According to one example aspect, a machine-learned hashing model that generates a binary hash for an input can be trained in conjunction with a machine-learned generative model that reconstructs the input from the binary hash. The present disclosure provides a novel generative approach to learn hash functions through Minimum Description Length principle such that the learned hash codes maximally compress the dataset. According to another example aspect, the present disclosure provides an efficient learning algorithm based on the stochastic distributional gradient, which avoids the notorious difficulty caused by binary output constraints, to jointly optimize the parameters of the hashing model and the associated generative model. The present disclosure also provides extensive experiments which show that the systems and methods described herein achieve better retrieval results than the existing state-of-the-art methods.

公开(公告)号：US20220343152A1

公开(公告)日：2022-10-27

申请号：US17239320

申请日：2021-04-23

Applicant: Google LLC

Inventor： Bo Dai ,  Mengjiao Yang ,  Hanjun Dai ,  Dale Eric Schuurmans

IPC: G06N3/08 ,  G06N3/04

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generative modelling of an exchangeable sets. Methods can include obtaining a dataset of training observations. Each training observation is an exchangeable set that includes a plurality of data points. Each training observations is processed using a first neural network to generate parameters of a first probability distribution based on which a latent variable is sampled. The latent variable is processed using a second neural network to generate a new observation that includes a plurality of data points. The training observation and the new observation is processed using an energy neural network to generate an estimate of an energy of the training observation and the new observation. The energy neural network is then trained to optimize an objective function that measures the difference between the estimate of the energy of the training observation and the new observation.