公开(公告)号：US20240273294A1

公开(公告)日：2024-08-15

申请号：US18166806

申请日：2023-02-09

Applicant: Google LLC

Inventor： Siamak Shakeri ,  Cicero Nogueira dos Santos ,  Daniel Matthew Cer ,  Zhe Dong ,  Jianmo Ni ,  Yun-Hsuan Sung ,  John Nham

IPC: G06F40/295 ,  G06N3/0455 ,  G06N3/084

CPC classification number: G06F40/295 ,  G06N3/0455 ,  G06N3/084

Abstract: The technology employs soft knowledge prompts (KPs) to inject relevant world knowledge into language models. This includes training KPs via self-supervised learning on data from one or more knowledge bases. KPs are task independent and can function as an external memory of the language models. KPs may be entity-centric, meaning that each prompt primarily encodes information about one entity from a given knowledge base. A method includes identifying a KP in response to a received input text, concatenating that KP to a sequence of word embeddings of the input text, applying the concatenated information to a trained language model, predicting an object entity name, computing a cross-entropy loss, and updating the identified KP based on the computed cross-entropy loss.

公开(公告)号：US20240232637A9

公开(公告)日：2024-07-11

申请号：US18491877

申请日：2023-10-23

Applicant: Google LLC

Inventor： Krishna Pragash Srinivasan ,  Michael Bendersky ,  Anupam Samanta ,  Lingrui Liao ,  Luca Bertelli ,  Ming-Wei Chang ,  Iftekhar Naim ,  Siddhartha Brahma ,  Siamak Shakeri ,  Hongkun Yu ,  John Nham ,  Karthik Raman ,  Raphael Dominik Hoffmann

IPC: G06N3/0895 ,  G06F16/903 ,  G06F16/93 ,  G06N3/0455

CPC classification number: G06N3/0895 ,  G06F16/90335 ,  G06F16/93 ,  G06N3/0455

Abstract: Provided are computing systems, methods, and platforms that train query processing models, such as large language models, to perform query intent classification tasks by using retrieval augmentation and multi-stage distillation. Unlabeled training examples of queries may be obtained, and a set of the training examples may be augmented with additional feature annotations to generate augmented training examples. A first query processing model may annotate the retrieval augmented queries to generate inferred labels for the augmented training examples. A second query processing model may be trained on the inferred labels, distilling the query processing model that was trained with retrieval augmentation into a non-retrieval augmented query processing model. The second query processing model may annotate the entire set of unlabeled training examples. Another stage of distillation may train a third query processing model using the entire set of unlabeled training examples without retrieval augmentation.

公开(公告)号：US20240135187A1

公开(公告)日：2024-04-25

申请号：US18491877

申请日：2023-10-22

Applicant: Google LLC

Inventor： Krishna Pragash Srinivasan ,  Michael Bendersky ,  Anupam Samanta ,  Lingrui Liao ,  Luca Bertelli ,  Ming-Wei Chang ,  Iftekhar Naim ,  Siddhartha Brahma ,  Siamak Shakeri ,  Hongkun Yu ,  John Nham ,  Karthik Raman ,  Raphael Dominik Hoffmann

IPC: G06N3/0895 ,  G06F16/903 ,  G06F16/93 ,  G06N3/0455

CPC classification number: G06N3/0895 ,  G06F16/90335 ,  G06F16/93 ,  G06N3/0455

Abstract: Provided are computing systems, methods, and platforms that train query processing models, such as large language models, to perform query intent classification tasks by using retrieval augmentation and multi-stage distillation. Unlabeled training examples of queries may be obtained, and a set of the training examples may be augmented with additional feature annotations to generate augmented training examples. A first query processing model may annotate the retrieval augmented queries to generate inferred labels for the augmented training examples. A second query processing model may be trained on the inferred labels, distilling the query processing model that was trained with retrieval augmentation into a non-retrieval augmented query processing model. The second query processing model may annotate the entire set of unlabeled training examples. Another stage of distillation may train a third query processing model using the entire set of unlabeled training examples without retrieval augmentation.