公开(公告)号：US11586904B2

公开(公告)日：2023-02-21

申请号：US16130058

申请日：2018-09-13

Applicant: Google LLC

Inventor： Sashank Jakkam Reddi ,  Sanjiv Kumar ,  Satyen Chandrakant Kale

IPC: G06N3/08 ,  G06N3/04 ,  G06F17/16 ,  G06N20/00

Abstract: Generally, the present disclosure is directed to systems and methods that perform adaptive optimization with improved convergence properties. The adaptive optimization techniques described herein are useful in various optimization scenarios, including, for example, training a machine-learned model such as, for example, a neural network. In particular, according to one aspect of the present disclosure, a system implementing the adaptive optimization technique can, over a plurality of iterations, employ an adaptive learning rate while also ensuring that the learning rate is non-increasing.

公开(公告)号：US20210064634A1

公开(公告)日：2021-03-04

申请号：US17001850

申请日：2020-08-25

Applicant: Google LLC

Inventor： Ruiqi Guo ,  David Simcha ,  Quan Geng ,  Felix Chern ,  Sanjiv Kumar ,  Xiang Wu

IPC: G06F16/25 ,  G06F16/2457 ,  H03M7/30

Abstract: Generally, the present disclosure is directed to systems and methods of quantizing a database with respect to a novel loss or quantization error function which applies a weight to an error measurement of quantized elements respectively corresponding to the datapoints in the database. The weight is determined based on the magnitude of an inner product between the respective datapoints and a query compared therewith. In contrast to previous work, embodiments of the proposed loss function are responsive to the expected magnitude of an inner product between the respective datapoints and a query compared therewith and can prioritize error reduction for higher-ranked pairings of the query and the datapoints. Thus, the systems and methods of the present disclosure provide solutions to some of the problems with traditional quantization approaches, which regard all error as equally impactful.

公开(公告)号：US20200104686A1

公开(公告)日：2020-04-02

申请号：US16586702

申请日：2019-09-27

Applicant: Google LLC

Inventor： Yang Li ,  Sanjiv Kumar ,  Pei-Hung Chen ,  Si Si ,  Cho-Jui Hsieh

IPC: G06N3/04 ,  G06K9/62 ,  G06F17/18 ,  G06F17/16

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for decreasing neural network inference times using softmax approximation. One of the methods includes maintaining data specifying a respective softmax weight vector for each output in a vocabulary of possible neural network outputs; receiving a neural network input; processing the neural network input using one or more initial neural network layers to generate a context vector for the neural network input; and generating an approximate score distribution over the vocabulary of possible neural network outputs for the neural network input, comprising: processing the context vector using a screening model configured to predict a proper subset of the vocabulary for the context input; and generating a respective logit for each output that is in the proper subset, comprising applying the softmax weight vector for the output to the context vector.

公开(公告)号：US10394777B2

公开(公告)日：2019-08-27

申请号：US14951909

申请日：2015-11-25

Applicant: Google LLC

Inventor： Xinnan Yu ,  Sanjiv Kumar ,  Ruiqi Guo

IPC: G06F17/00 ,  G06F16/22 ,  G06F16/33 ,  G06F16/951

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for efficiently performing linear projections. In one aspect, a method includes actions for obtaining a plurality of content items from one or more content sources. Additional actions include, extracting a plurality of features from each of the plurality of content items, generating a feature vector for each of the extracted features in order to create a search space, generating a series of element matrices based upon the generated feature vectors, transforming the series of element matrices into a structured matrix such that the transformation preserves one or more relationships associated with each element matrix of the series of element matrices, receiving a search object, searching the enhanced search space based on the received search object, provided one or more links to a content item that are responsive to the search object.

公开(公告)号：US20190114343A1

公开(公告)日：2019-04-18

申请号：US15783685

申请日：2017-10-13

Applicant: Google LLC

Inventor： Ruiqi Guo ,  Bo Dai ,  Sanjiv Kumar

IPC: G06F17/30 ,  G06F15/18

Abstract: The present disclosure provides systems and methods that perform stochastic generative hashing. According to one example aspect, a machine-learned hashing model that generates a binary hash for an input can be trained in conjunction with a machine-learned generative model that reconstructs the input from the binary hash. The present disclosure provides a novel generative approach to learn hash functions through Minimum Description Length principle such that the learned hash codes maximally compress the dataset. According to another example aspect, the present disclosure provides an efficient learning algorithm based on the stochastic distributional gradient, which avoids the notorious difficulty caused by binary output constraints, to jointly optimize the parameters of the hashing model and the associated generative model. The present disclosure also provides extensive experiments which show that the systems and methods described herein achieve better retrieval results than the existing state-of-the-art methods.

公开(公告)号：US12271810B2

公开(公告)日：2025-04-08

申请号：US17100253

申请日：2020-11-20

Applicant: Google LLC

Inventor： Sashank Jakkam Reddi ,  Sanjiv Kumar ,  Manzil Zaheer ,  Zachary Burr Charles ,  Zachary Alan Garrett ,  John Keith Rush ,  Jakub Konecny ,  Hugh Brendan McMahan

IPC: G06N3/08 ,  G06N3/04 ,  G06N3/045

Abstract: A computing system and method can be used to implement a version of federated learning (FL) that incorporates adaptivity (e.g., leverages an adaptive learning rate). In particular, the present disclosure provides a general optimization framework in which (1) clients perform multiple epochs of training using a client optimizer to minimize loss on their local data and (2) a server system updates its global model by applying a gradient-based server optimizer to the average of the clients' model updates. This framework can seamlessly incorporate adaptivity by using adaptive optimizers as client and/or server optimizers. Building upon this general framework, the present disclosure also provides example specific adaptive optimization techniques for FL which use per-coordinate methods as server optimizers. By focusing on adaptive server optimization, the use of adaptive learning rates is enabled without increase in client storage or communication costs and compatibility with cross-device FL can be ensured.

公开(公告)号：US20240098138A1

公开(公告)日：2024-03-21

申请号：US18240799

申请日：2023-08-31

Applicant: Google LLC

Inventor： Ananda Theertha Suresh ,  Sanjiv Kumar ,  Hugh Brendan McMahan ,  Xinnan Yu

IPC: H04L67/10 ,  G06F17/12 ,  G06F17/16 ,  G06F17/18 ,  G06N7/01 ,  G06N20/00 ,  H03M7/30 ,  H03M7/40

CPC classification number: H04L67/10 ,  G06F17/12 ,  G06F17/16 ,  G06F17/18 ,  G06N7/01 ,  G06N20/00 ,  H03M7/3059 ,  H03M7/3082 ,  H03M7/40 ,  H04L67/01

Abstract: The present disclosure provides systems and methods for communication efficient distributed mean estimation. In particular, aspects of the present disclosure can be implemented by a system in which a number of vectors reside on a number of different clients, and a centralized server device seeks to estimate the mean of such vectors. According to one aspect of the present disclosure, a client computing device can rotate a vector by a random rotation matrix and then subsequently perform probabilistic quantization on the rotated vector. According to another aspect of the present disclosure, subsequent to quantization but prior to transmission, the client computing can encode the quantized vector according to a variable length coding scheme (e.g., by computing variable length codes).

公开(公告)号：US20230394310A1

公开(公告)日：2023-12-07

申请号：US18453837

申请日：2023-08-22

Applicant: Google LLC

Inventor： Sashank Jakkam Reddi ,  Sanjiv Kumar ,  Manzil Zaheer ,  Satyen Chandrakant Kale

IPC: G06N3/08 ,  G06N3/045

CPC classification number: G06N3/08 ,  G06N3/045

Abstract: Generally, the present disclosure is directed to systems and methods that perform adaptive optimization with improved convergence properties. The adaptive optimization techniques described herein are useful in various optimization scenarios, including, for example, training a machine-learned model such as, for example, a neural network. In particular, according to one aspect of the present disclosure, a system implementing the adaptive optimization technique can, over a plurality of iterations, employ an adaptive effective learning rate while also ensuring that the effective learning rate is non-increasing.

公开(公告)号：US20230017505A1

公开(公告)日：2023-01-19

申请号：US17375960

申请日：2021-07-14

Applicant: Google LLC

Inventor： Aditya Krishna Menon ,  Sanjiv Kumar ,  Himanshu Jain ,  Andreas Veit ,  Ankit Singh Rawat ,  Gayan Sadeep Jayasumana Hirimbura Matara Kankanamge

IPC: G06N3/08 ,  G06K9/62

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for accounting for long-tail training data.

公开(公告)号：US20220335274A1

公开(公告)日：2022-10-20

申请号：US17721292

申请日：2022-04-14

Applicant: Google LLC

Inventor： Ankit Singh Rawat ,  Manzil Zaheer ,  Aditya Krishna Menon ,  Sanjiv Kumar ,  Amr Ahmed

IPC: G06N3/04 ,  G06K9/62

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for multi-stage computationally-efficient inference using a first and second neural network.