公开(公告)号：US20220335274A1

公开(公告)日：2022-10-20

申请号：US17721292

申请日：2022-04-14

Applicant: Google LLC

Inventor： Ankit Singh Rawat ,  Manzil Zaheer ,  Aditya Krishna Menon ,  Sanjiv Kumar ,  Amr Ahmed

IPC: G06N3/04 ,  G06K9/62

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for multi-stage computationally-efficient inference using a first and second neural network.

公开(公告)号：US12147794B2

公开(公告)日：2024-11-19

申请号：US18070015

申请日：2022-11-28

Applicant: Google LLC

Inventor： Joey Hong ,  Rishabh Singh ,  Joel Galenson ,  Jonathan Malmaud ,  Manzil Zaheer

IPC: G06F8/51

Abstract: Implementations are described herein for predicting symbolic transformation templates to automate source code transformations. In various implementations, pair(s) of predecessor and successor source code snippets may be processed using a symbolic transformation template prediction (STTP) model to predict a symbolic transformation template that includes a predecessor portion that matches the predecessor source code snippet(s) of the pair(s) and a successor portion that matches the successor source code snippet(s) of the pair(s). At least one additional predecessor source code snippet may be identified that matches the predecessor portion of the predicted symbolic transformation template. Placeholders of the predecessor portion of the predicted symbolic transformation template may be bound to one or more tokens of the at least one additional predecessor source code snippet to create binding(s). The successor portion of the predicted symbolic transformation template may be applied to the bindings to generate additional successor source code snippet(s).

公开(公告)号：US20230325164A1

公开(公告)日：2023-10-12

申请号：US17717609

申请日：2022-04-11

Applicant: Google LLC

Inventor： Rishabh Singh ,  Manzil Zaheer

IPC: G06F8/51 ,  G06F8/73 ,  G06K9/62

CPC classification number: G06F8/51 ,  G06F8/73 ,  G06K9/6223

Abstract: Techniques are described herein for translating a source code snippet from a first programming language to a second programming language independently of sequence-to-sequence decoding. In various implementations, the source code snippet written in the first programming language may be processed using an encoder portion of a transformer network to generate an embedding of the source code snippet. The embedding of the source code snippet may be processed using an all-pair attention layer to generate an attended embedding of the source code snippet. The attended embedding of the source code snippet may be processed using an output layer to generate, by way of a single transformation of the attended embedding of the source code snippet, data indicative of a translation of the source code snippet in the second programming language.

公开(公告)号：US11238332B2

公开(公告)日：2022-02-01

申请号：US17341193

申请日：2021-06-07

Applicant: Google LLC

Inventor： Joshua Timothy Ainslie ,  Santiago Ontañón ,  Philip Pham ,  Manzil Zaheer ,  Guru Guruganesh ,  Kumar Avinava Dubey ,  Amr Ahmed

IPC: G06N3/04 ,  G06N3/08 ,  G06N3/063 ,  G06N20/00

Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing network inputs using an attention neural network that has one or more sparse attention sub-layers. Each sparse attention sub-layer is configured to apply a sparse attention mechanism that attends differently for input positions that are in a first proper subset of the input positions in the input to the sub-layer than for positions that are not in the first proper subset.

公开(公告)号：US20210073639A1

公开(公告)日：2021-03-11

申请号：US17100253

申请日：2020-11-20

Applicant: Google LLC

Inventor： Sashank Jakkam Reddi ,  Sanjiv Kumar ,  Manzil Zaheer ,  Zachary Charles ,  Zach Garrett ,  Keith Rush ,  Jakub Konecny ,  Hugh Brendan McMahan

IPC: G06N3/08 ,  G06N3/04

Abstract: A computing system and method can be used to implement a version of federated learning (FL) that incorporates adaptivity (e.g., leverages an adaptive learning rate). In particular, the present disclosure provides a general optimization framework in which (1) clients perform multiple epochs of training using a client optimizer to minimize loss on their local data and (2) a server system updates its global model by applying a gradient-based server optimizer to the average of the clients' model updates. This framework can seamlessly incorporate adaptivity by using adaptive optimizers as client and/or server optimizers. Building upon this general framework, the present disclosure also provides example specific adaptive optimization techniques for FL which use per-coordinate methods as server optimizers. By focusing on adaptive server optimization, the use of adaptive learning rates is enabled without increase in client storage or communication costs and compatibility with cross-device FL can be ensured.

公开(公告)号：US20200175365A1

公开(公告)日：2020-06-04

申请号：US16657356

申请日：2019-10-18

Applicant: Google LLC

Inventor： Sashank Jakkam Reddi ,  Sanjiv Kumar ,  Manzil Zaheer ,  Satyen Chandrakant Kale

IPC: G06N3/08 ,  G06N3/04

Abstract: Generally, the present disclosure is directed to systems and methods that perform adaptive optimization with improved convergence properties. The adaptive optimization techniques described herein are useful in various optimization scenarios, including, for example, training a machine-learned model such as, for example, a neural network. In particular, according to one aspect of the present disclosure, a system implementing the adaptive optimization technique can, over a plurality of iterations, employ an adaptive effective learning rate while also ensuring that the effective learning rate is non-increasing.

公开(公告)号：US20250111210A1

公开(公告)日：2025-04-03

申请号：US18900531

申请日：2024-09-27

Applicant: Google LLC

Inventor： Chong You ,  Guru Guruganesh ,  Joshua Timothy Ainslie ,  Manzil Zaheer ,  Sanjiv Kumar ,  Santiago Ontañón ,  Shanda Li ,  Venkata Sesha Pavana Srinadh Bhojanapalli ,  Sumit Sanghai

IPC: G06N3/0475

Abstract: Systems and methods for processing inputs using attention neural networks. In particular, one or more of the attention layers within the attention neural network compute relative position biases using functional interpolation.

公开(公告)号：US20240311405A1

公开(公告)日：2024-09-19

申请号：US18337316

申请日：2023-06-19

Applicant: GOOGLE LLC

Inventor： Seungyeon Kim ,  Ankit Singh Rawat ,  Wittawat Jitkrittum ,  Hari Narasimhan ,  Sashank Reddi ,  Neha Gupta ,  Srinadh Bhojanapalli ,  Aditya Menon ,  Manzil Zaheer ,  Tal Schuster ,  Sanjiv Kumar ,  Toby Boyd ,  Zhifeng Chen ,  Emanuel Taropa ,  Vikram Kasivajhula ,  Trevor Strohman ,  Martin Baeuml ,  Leif Schelin ,  Yanping Huang

IPC: G06F16/332

CPC classification number: G06F16/3329

Abstract: Implementations disclose selecting, in response to receiving a request and from among multiple candidate generative models (e.g., multiple candidate large language models (LLMs)) with differing computational efficiencies, a particular generative model to utilize in generating a response to the request. Those implementations reduce latency and/or conserve computational resource(s) through selection, for various requests, of a more computationally efficient generative model for utilization in lieu of a less computationally efficient generative model. Further, those implementations seek to achieve such benefits, through utilization of more computationally efficient generative models, while also still selectively utilizing less computationally efficient generative models for certain requests to mitigate occurrences of a generated response being inaccurate and/or under-specified. This, in turn, can mitigate occurrences of computational and/or network inefficiencies that result from a user issuing a follow-up request to cure the inaccuracies and/or under-specification of a generated response.

公开(公告)号：US12014160B2

公开(公告)日：2024-06-18

申请号：US17717609

申请日：2022-04-11

Applicant: Google LLC

Inventor： Rishabh Singh ,  Manzil Zaheer

IPC: G06F8/51 ,  G06F8/73 ,  G06F18/23213

CPC classification number: G06F8/51 ,  G06F8/73 ,  G06F18/23213

Abstract: Techniques are described herein for translating a source code snippet from a first programming language to a second programming language independently of sequence-to-sequence decoding. In various implementations, the source code snippet written in the first programming language may be processed using an encoder portion of a transformer network to generate an embedding of the source code snippet. The embedding of the source code snippet may be processed using an all-pair attention layer to generate an attended embedding of the source code snippet. The attended embedding of the source code snippet may be processed using an output layer to generate, by way of a single transformation of the attended embedding of the source code snippet, data indicative of a translation of the source code snippet in the second programming language.

公开(公告)号：US11693637B1

公开(公告)日：2023-07-04

申请号：US17319739

申请日：2021-05-13

Applicant: Google LLC

Inventor： Rishabh Singh ,  Hanjun Dai ,  Manzil Zaheer ,  Artem Goncharuk ,  Karen Davis ,  David Andre

IPC: G06F8/30 ,  G06F8/41 ,  G06F40/40 ,  G06N3/08 ,  G06F40/279 ,  G06N7/01

CPC classification number: G06F8/436 ,  G06F40/279 ,  G06F40/40 ,  G06N3/08 ,  G06N7/01

Abstract: Using a natural language (NL) latent presentation in the automated conversion of source code from a base programming language (e.g., C++) to a target programming language (e.g., Python). A base-to-NL model can be used to generate an NL latent representation by processing a base source code snippet in the base programming language. Further, an NL-to-target model can be used to generate a target source code snippet in the target programming language (that is functionally equivalent to the base source code snippet), by processing the NL latent representation. In some implementations, output(s) from the NL-to-target model indicate canonical representation(s) of variables, and in generating the target source code snippet, technique(s) are used to match those canonical representation(s) to variable(s) of the base source code snippet. In some implementations, multiple candidate target source code snippets are generated, and a subset (e.g., one) is selected based on evaluation(s).