公开(公告)号：US20240330762A1

公开(公告)日：2024-10-03

申请号：US18293638

申请日：2021-09-03

Applicant: Google LLC

Inventor： Dana Alon ,  Yair Alon ,  Elad Eban

IPC: G06N20/00

CPC classification number: G06N20/00

Abstract: Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for improving the representation of items of a vocabulary in an embedding space for use in machine learning models. An embedding matrix is generated wherein each row in the embedding matrix is a vector of elements and corresponds to an item of a vocabulary. A score is assigned to each vector in the embedding matrix indicating a probability of its corresponding vector being used in the machine learning model. The scores are iteratively updated by sampling a proper subset of vectors and updating the elements of each respective vector in the proper subset of vectors based on the respective scores of vectors. The score of each vector are then updated based on a loss function of the machine learning model. The embedding matrix is then re-structured based on the updated scores of the vectors.

公开(公告)号：US20240370717A1

公开(公告)日：2024-11-07

申请号：US18313189

申请日：2023-05-05

Applicant: Google LLC

Inventor： Qifei Wang ,  Yicheng Fan ,  Wei Xu ,  Jiayu Ye ,  Lu Wang ,  Chuo-Ling Chang ,  Dana Alon ,  Erik Nathan Vee ,  Hongkun Yu ,  Matthias Grundmann ,  Shanmugasundaram Ravikumar ,  Andrew Stephen Tomkins

IPC: G06N3/08

Abstract: A method for a cross-platform distillation framework includes obtaining a plurality of training samples. The method includes generating, using a student neural network model executing on a first processing unit, a first output based on a first training sample. The method also includes generating, using a teacher neural network model executing on a second processing unit, a second output based on the first training sample. The method includes determining, based on the first output and the second output, a first loss. The method further includes adjusting, based on the first loss, one or more parameters of the student neural network model. The method includes repeating the above steps for each training sample of the plurality of training samples.

公开(公告)号：US20240232686A1

公开(公告)日：2024-07-11

申请号：US18012292

申请日：2022-07-29

Applicant: Google LLC

Inventor： Yicheng Fan ,  Jingyue Shen ,  Deqiang Chen ,  Peter Shaosen Young ,  Dana Alon ,  Erik Nathan Vee ,  Shanmugasundaram Ravikumar ,  Andrew Tomkins

IPC: G06N20/00

CPC classification number: G06N20/00

Abstract: Systems and methods of the present disclosure are directed to portion-specific compression and optimization of machine-learned models. For example, a method for portion-specific compression and optimization of machine-learned models includes obtaining data descriptive of one or more respective sets of compression schemes for one or more model portions of a plurality of model portions of a machine-learned model. The method includes evaluating a cost function to respectively select one or more candidate compression schemes from the one or more sets of compression schemes. The method includes respectively applying the one or more candidate compression schemes to the one or more model portions to obtain a compressed machine-learned model comprising one or more compressed model portions that correspond to the one or more model portions.