-
公开(公告)号：US20250045316A1
公开(公告)日：2025-02-06
申请号：US18788178
申请日：2024-07-30
Applicant: Google LLC
Inventor： Jinhyuk Lee , Zhuyun Dai , Xiaoqi Ren , Iftekhar Naim , Yi Luan , Blair Yuxin Chen , Siddhartha Reddy Jonnalagadda , Ming-Wei Chang , Daniel Matthew Cer , Gustavo Adolfo Hernandez Abrego , Jeremy Robert Cole , Colin Hearne Evans , Yuzhe Zhao , Pranay Bhatia , Rajvi Kapadia , Riham Hassan Abdel-Moneim Mansour , Raphael Dominik Hoffman , Simon Kunio Tokumine , Scott Bradley Huffman , Stephen Zachary Karukas , Michael Yiupun Kwong , Shu Zheng , Yan Qiao , Lukas Rutishauser , Anand Rajan Iyer
Abstract: An example method includes providing, to a sequence model (i) a plurality of few-shot prompts, wherein each prompt comprises a demonstration passage, a demonstration task, and a demonstration query, wherein the demonstration task describes a type of retrieval, and wherein the demonstration query is relevant to the demonstration task, and (ii) a plurality of passages sampled from a corpus of passages. The method also includes receiving, from the sequence model and for the plurality of passages and based on the plurality of few-shot prompts, a respective plurality of predicted task-query pairs, the sequence model having been prompted to predict a task based on an input passage, and predict an output query relevant to the predicted task. The method further includes generating a synthetic training dataset comprising the plurality of passages and the respective plurality of predicted task-query pairs. The method also includes providing the synthetic training dataset.
-
公开(公告)号：US11003865B1
公开(公告)日：2021-05-11
申请号：US16879457
申请日：2020-05-20
Applicant: Google LLC
Inventor： Kenton Chiu Tsun Lee , Kelvin Gu , Zora Tung , Panupong Pasupat , Ming-Wei Chang
Abstract: Systems and methods for pre-training and fine-tuning of neural-network-based language models are disclosed in which a neural-network-based textual knowledge retriever is trained along with the language model. In some examples, the knowledge retriever obtains documents from an unlabeled pre-training corpus, generates its own training tasks, and learns to retrieve documents relevant to those tasks. In some examples, the knowledge retriever is further refined using supervised open-QA questions. The framework of the present technology provides models that can intelligently retrieve helpful information from a large unlabeled corpus, rather than requiring all potentially relevant information to be stored implicitly in the parameters of the neural network. This framework may thus reduce the storage space and complexity of the neural network, and also enable the model to more effectively handle new tasks that may be different than those on which it was pre-trained.
-
公开(公告)号：US20240232637A9
公开(公告)日：2024-07-11
申请号：US18491877
申请日：2023-10-23
Applicant: Google LLC
Inventor： Krishna Pragash Srinivasan , Michael Bendersky , Anupam Samanta , Lingrui Liao , Luca Bertelli , Ming-Wei Chang , Iftekhar Naim , Siddhartha Brahma , Siamak Shakeri , Hongkun Yu , John Nham , Karthik Raman , Raphael Dominik Hoffmann
IPC: G06N3/0895 , G06F16/903 , G06F16/93 , G06N3/0455
CPC classification number: G06N3/0895 , G06F16/90335 , G06F16/93 , G06N3/0455
Abstract: Provided are computing systems, methods, and platforms that train query processing models, such as large language models, to perform query intent classification tasks by using retrieval augmentation and multi-stage distillation. Unlabeled training examples of queries may be obtained, and a set of the training examples may be augmented with additional feature annotations to generate augmented training examples. A first query processing model may annotate the retrieval augmented queries to generate inferred labels for the augmented training examples. A second query processing model may be trained on the inferred labels, distilling the query processing model that was trained with retrieval augmentation into a non-retrieval augmented query processing model. The second query processing model may annotate the entire set of unlabeled training examples. Another stage of distillation may train a third query processing model using the entire set of unlabeled training examples without retrieval augmentation.
-
公开(公告)号：US20240135187A1
公开(公告)日：2024-04-25
申请号：US18491877
申请日：2023-10-22
Applicant: Google LLC
Inventor： Krishna Pragash Srinivasan , Michael Bendersky , Anupam Samanta , Lingrui Liao , Luca Bertelli , Ming-Wei Chang , Iftekhar Naim , Siddhartha Brahma , Siamak Shakeri , Hongkun Yu , John Nham , Karthik Raman , Raphael Dominik Hoffmann
IPC: G06N3/0895 , G06F16/903 , G06F16/93 , G06N3/0455
CPC classification number: G06N3/0895 , G06F16/90335 , G06F16/93 , G06N3/0455
Abstract: Provided are computing systems, methods, and platforms that train query processing models, such as large language models, to perform query intent classification tasks by using retrieval augmentation and multi-stage distillation. Unlabeled training examples of queries may be obtained, and a set of the training examples may be augmented with additional feature annotations to generate augmented training examples. A first query processing model may annotate the retrieval augmented queries to generate inferred labels for the augmented training examples. A second query processing model may be trained on the inferred labels, distilling the query processing model that was trained with retrieval augmentation into a non-retrieval augmented query processing model. The second query processing model may annotate the entire set of unlabeled training examples. Another stage of distillation may train a third query processing model using the entire set of unlabeled training examples without retrieval augmentation.
-
-
-
Search Results
4
records
(took 0.013 seconds)
