-
公开(公告)号：US11551159B2
公开(公告)日：2023-01-10
申请号：US16724604
申请日：2019-12-23
Applicant: Google LLC
Inventor： Abhinav Kumar Rastogi , Raghav Gupta , Xiaoxue Zang , Srinivas Kumar Sunkara , Pranav Khaitan
Abstract: Generally, the present disclosure is directed to systems and methods for performing task-oriented response generation that can provide advantages for artificial intelligence systems or other computing systems that include natural language processing for interpreting user input. Example implementations can process natural language descriptions of various services that can be accessed by the system. In response to a natural language input, systems can identify relevant values for executing one of the service(s), based in part on comparing embedded representations of the natural language input and the natural language description using a machine learned model.
-
公开(公告)号：US20210192397A1
公开(公告)日：2021-06-24
申请号：US16724604
申请日：2019-12-23
Applicant: Google LLC
Inventor： Abhinav Kumar Rastogi , Raghav Gupta , Xiaoxue Zang , Srinivas Kumar Sunkara , Pranav Khaitan
Abstract: Generally, the present disclosure is directed to systems and methods for performing task-oriented response generation that can provide advantages for artificial intelligence systems or other computing systems that include natural language processing for interpreting user input. Example implementations can process natural language descriptions of various services that can be accessed by the system. In response to a natural language input, systems can identify relevant values for executing one of the service(s), based in part on comparing embedded representations of the natural language input and the natural language description using a machine learned model.
-
公开(公告)号：US11797862B2
公开(公告)日：2023-10-24
申请号：US16749570
申请日：2020-01-22
Applicant: Google LLC
Inventor： Yang Song , Raghav Gupta , Dengyong Zhou , Sanqiang Zhao
IPC: G06N3/088 , G06F40/284 , G06N3/045
CPC classification number: G06N3/088 , G06F40/284 , G06N3/045
Abstract: Provided is a knowledge distillation technique for training a student language model that, relative to a larger teacher language model, has a significantly smaller vocabulary, lower embedding dimensions, and/or hidden state dimensions. Specifically, aspects of the present disclosure are directed to a dual-training mechanism that trains the teacher and student language models simultaneously to obtain optimal word embeddings for the student vocabulary. In some implementations, this approach can be combined with learning shared projection matrices that transfer layer-wise knowledge from the teacher language model to the student language model. Example experimental results have also demonstrated higher compression efficiency and accuracy when compared with other state-of-the-art compression techniques, including the ability to compress the BERTBASE model by more than 60×, with only a minor drop in downstream task metrics, resulting in a language model with a footprint of under 7 MB.
-
公开(公告)号：US12087288B2
公开(公告)日：2024-09-10
申请号：US17273555
申请日：2019-09-04
Applicant: Google LLC
Inventor： Dilek Hakkani-Tur , Abhinav Kumar Rastogi , Raghav Gupta
IPC: G10L15/18 , G06F40/117 , G06F40/284 , G06F40/35 , G06N3/02 , G10L15/16 , G10L15/22
CPC classification number: G10L15/1815 , G06F40/117 , G06F40/284 , G06F40/35 , G06N3/02 , G10L15/16 , G10L15/22
Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media for dialogue systems. A transcription of a user utterance is obtained. The transcription of the utterance is tokenized to identify multiple tokens for the utterance. Token-level utterance encodings corresponding to different tokens of the transcription are generated. A system action encoding from data indicating system actions previously performed by the dialogue system are generated. A dialogue context vector based on the utterance encoding and the system action encoding are generated. The token-level utterance encodings, the system action encoding, and the dialogue context vector are processed using a slot tagger to produce token-level output vectors. A limited set of candidate token classifications for the tokens of the user utterance are determined based on the token-level utterance encodings. A response for output is provided in response to the user utterance.
-
公开(公告)号：US20240221731A1
公开(公告)日：2024-07-04
申请号：US18148037
申请日：2022-12-29
Applicant: Google LLC
Inventor： Raghav Gupta , Yuan Cao , Abhinav Kumar Rastogi , Harrison J. Lee , Jeffrey Liangjie Zhao
CPC classification number: G10L15/1815 , G06F40/35 , G10L15/063 , G10L2015/0633
Abstract: Example methods include determining an input prompt comprising an utterance labeled with a sequence of slot-value pairs, wherein the sequence of slot-value pairs indicates possible slots and values in the utterance, and wherein the utterance relates to a task. The methods include determining a contextual representation comprising a concatenation of a history of utterances exchanged between a user and a service agent. The utterances describe a context for the task. The methods include training, based on a concatenation of the input prompt and the contextual representation, a sequence-to-sequence language model to predict a sequence of dialog states for an input task. The sequence of dialog states comprise an assignment of values to slots for which the user has indicated a preference in dialog sequences. The methods include providing the trained sequence-to-sequence language model.
-
Patent Agency Ranking
公开(公告)号：US20210224660A1
公开(公告)日：2021-07-22
申请号：US16749570
申请日：2020-01-22
Applicant: Google LLC
Inventor： Yang Song , Raghav Gupta , Dengyong Zhou , Sanqiang Zhao
IPC: G06N3/08 , G06N3/04 , G06F40/284
Abstract: Provided is a knowledge distillation technique for training a student language model that, relative to a larger teacher language model, has a significantly smaller vocabulary, lower embedding dimensions, and/or hidden state dimensions. Specifically, aspects of the present disclosure are directed to a dual-training mechanism that trains the teacher and student language models simultaneously to obtain optimal word embeddings for the student vocabulary. In some implementations, this approach can be combined with learning shared projection matrices that transfer layer-wise knowledge from the teacher language model to the student language model. Example experimental results have also demonstrated higher compression efficiency and accuracy when compared with other state-of-the-art compression techniques, including the ability to compress the BERTBAsE model by more than 60×, with only a minor drop in downstream task metrics, resulting in a language model with a footprint of under 7 MB.
-
公开(公告)号：US20240220732A1
公开(公告)日：2024-07-04
申请号：US18148045
申请日：2022-12-29
Applicant: Google LLC
Inventor： Raghav Gupta , Yuan Cao , Abhinav Kumar Rastogi , Harrison J. Lee , Jeffrey Liangjie Zhao
CPC classification number: G06F40/35 , G06F16/367
Abstract: Example methods include determining an input schema representation for a task. The schema representation comprises natural language descriptions of slot and intent descriptions, wherein respective indices are associated with each of the slot descriptions and each of the intent descriptions. The methods include determining a contextual representation comprising a concatenation of a history of dialog sequences exchanged between a user and a service agent, wherein the dialog sequences describe a context for the task. The methods include training, a sequence-to-sequence language model and based on a concatenation of the input schema representation and the contextual representation, to predict a sequence of dialog states for an input task, wherein the sequence of dialog states comprises an assignment of values to slots for which the user has indicated a preference in dialog sequences corresponding to the input task. The methods include providing the trained sequence-to-sequence language model.
-
公开(公告)号：US20240013059A1
公开(公告)日：2024-01-11
申请号：US18471866
申请日：2023-09-21
Applicant: Google LLC
Inventor： Yang Song , Raghav Gupta , Dengyong Zhou , Sanqiang Zhao
IPC: G06N3/0455 , G06F40/40 , G06N3/08
CPC classification number: G06N3/0455 , G06F40/40 , G06N3/08 , G06F40/284
Abstract: Provided is a knowledge distillation technique for training a student language model that, relative to a larger teacher language model, has a significantly smaller vocabulary, lower embedding dimensions, and/or hidden state dimensions. Specifically, aspects of the present disclosure are directed to a dual-training mechanism that trains the teacher and student language models simultaneously to obtain optimal word embeddings for the student vocabulary. In some implementations, this approach can be combined with learning shared projection matrices that transfer layer-wise knowledge from the teacher language model to the student language model. Example experimental results have also demonstrated higher compression efficiency and accuracy when compared with other state-of-the-art compression techniques, including the ability to compress the BERTBASE model by more than 60×, with only a minor drop in downstream task metrics, resulting in a language model with a footprint of under 7 MB.
-
公开(公告)号：US20210217408A1
公开(公告)日：2021-07-15
申请号：US17273555
申请日：2019-09-04
Applicant: Google LLC
Inventor： Dilek Hakkani-Tur , Abhinav Kumar Rastogi , Raghav Gupta
IPC: G10L15/18 , G06F40/117 , G06F40/35 , G10L15/22 , G10L15/16 , G06F40/284 , G06N3/02
Abstract: Methods, systems, and apparatus, including computer programs encoded on computer storage media for dialogue systems. A transcription of a user utterance is obtained. The transcription of the utterance is tokenized to identify multiple tokens for the utterance. Token-level utterance encodings corresponding to different tokens of the transcription are generated. A system action encoding from data indicating system actions previously performed by the dialogue system are generated. A dialogue context vector based on the utterance encoding and the system action encoding are generated. The token-level utterance encodings, the system action encoding, and the dialogue context vector are processed using a slot tagger to produce token-level output vectors. A limited set of candidate token classifications for the tokens of the user utterance are determined based on the token-level utterance encodings. A response for output is provided in response to the user utterance.
-
公开(公告)号：US12260340B2
公开(公告)日：2025-03-25
申请号：US18471866
申请日：2023-09-21
Applicant: Google LLC
Inventor： Yang Song , Raghav Gupta , Dengyong Zhou , Sanqiang Zhao
IPC: G06N3/088 , G06F40/284 , G06N3/045
Abstract: Provided is a knowledge distillation technique for training a student language model that, relative to a larger teacher language model, has a significantly smaller vocabulary, lower embedding dimensions, and/or hidden state dimensions. Specifically, aspects of the present disclosure are directed to a dual-training mechanism that trains the teacher and student language models simultaneously to obtain optimal word embeddings for the student vocabulary. In some implementations, this approach can be combined with learning shared projection matrices that transfer layer-wise knowledge from the teacher language model to the student language model. Example experimental results have also demonstrated higher compression efficiency and accuracy when compared with other state-of-the-art compression techniques, including the ability to compress the BERTBASE model by more than 60×, with only a minor drop in downstream task metrics, resulting in a language model with a footprint of under 7 MB.
-
-
-
-
-
-
-
-
-
Search Results
11
records
(took 0.087 seconds)
