公开(公告)号：US20240378394A1

公开(公告)日：2024-11-14

申请号：US18231650

申请日：2023-08-08

Applicant: GOOGLE LLC

Inventor： Ragha Kotikalapudi ,  Chen Zhu ,  Steven Zheng ,  Sahitya Potluri ,  Yu Du ,  Heng-Tze Cheng ,  Quoc Le ,  Ed H. Chi

IPC: G06F40/40 ,  G06N3/09

Abstract: Implementations described herein relate to using self-evaluation when utilizing a large language model (LLM) to generate a response to a natural language (NL) based input. The LLM can be used to process the NL based input to generate a plurality of responses, and to generate a critique of those responses by comparing the responses to a set of response evaluation criteria. One of the responses can then be selected based on the comparison with the set of response evaluation criteria which can vary from one NL based input to another. If the NL based input was obtained a user of a client device during an inference stage, then the selected response can be rendered for presentation to the user. If the NL based input was obtained during a training stage, then the selected response can be stored as a training instance and optionally in association with additional data.

公开(公告)号：US20250086405A1

公开(公告)日：2025-03-13

申请号：US18481803

申请日：2023-10-05

Applicant: GOOGLE LLC

Inventor： Swaroop Mishra ,  Ragha Kotikalapudi ,  Obaid Sarvana ,  Sahitya Potluri ,  YaGuang Li ,  Taylor Bos ,  Steven Zheng ,  Hanzhao Lin ,  Chenkai Kuang ,  Heng-Tze Cheng ,  Ed H. Chi ,  Quoc Le

IPC: G06F40/40 ,  G06N3/08

Abstract: Some implementations relate to generating a training and/or evaluation dataset with LLM prompts (e.g., derived from user queries) based on a prompt complexity. An input prompt, for example derived from a user query, is received. The input prompt is decomposed into a prompt tree comprising a plurality of nodes. The plurality of nodes comprise: a plurality of leaf nodes corresponding to simple sub-prompts of the input query; a plurality of branch nodes of sub-prompts each corresponding to multiple simple sub-prompts; and a root node corresponding to the input prompt. A prompt complexity is determined based on a path length of the prompt tree. The prompt complexity is compared to a threshold complexity. If the prompt complexity is above the threshold complexity, the input prompt is included in a set of training prompts and/or a set of evaluation prompts.

公开(公告)号：US20250045534A1

公开(公告)日：2025-02-06

申请号：US18378434

申请日：2023-10-10

Applicant: GOOGLE LLC

Inventor： Swaroop Mishra ,  Ragha Kotikalapudi ,  Sahitya Potluri ,  Taylor Bos ,  YaGuang Li ,  Hanzhao Lin ,  Steven Zheng ,  Yu Du ,  Chen Zhu ,  Chenkai Kuang ,  Xinying Song ,  Heng-Tze Cheng ,  Ed H. Chi ,  Quoc Le

IPC: G06F40/40

Abstract: Implementations relate to a method implemented by one or more processors, the method including: receiving natural language (NL) based input associated with a client device; generating, using a large language model (LLM) and based on processing the NL based input, LLM output; determining, based on the LLM output, a sequence of LLM responses, the sequence of LLM responses including at least one intermediate LLM response and a final LLM response. In some implementations, the method may further include causing the final LLM response to be rendered at the client device. In additional or alternative implementations, the method may further include storing, as an instance of training data for fine-tuning the LLM or an additional LLM, the NL based input along with the final LLM response.

公开(公告)号：US20240394471A1

公开(公告)日：2024-11-28

申请号：US18231586

申请日：2023-08-08

Applicant: GOOGLE LLC

Inventor： Ragha Kotikalapudi ,  Swaroop Mishra ,  Sahitya Potluri ,  Taylor Bos ,  Yu Du ,  Chen Zhu ,  Steven Zheng ,  Hanzhao Lin ,  Summer Yue ,  Heng-Tze Cheng ,  Quoc Le ,  Ed H. Chi

IPC: G06F40/20

Abstract: Implementations relate to improving instruction following capabilities of large language models (LLMs) using instruction decomposition, self-evaluation, and optionally progressive refinement. Processor(s) of a system can: obtain natural language (NL) based input, generate a plurality of candidate responses and evaluate the candidate responses based on instructions included in the NL based input, using an LLM, and progressively refine the candidate responses until it is determined that one or more termination criteria are satisfied. In some implementations, the NL based input can be received from a client device. In these implementations, a given candidate response that is progressively refined can be rendered for presentation at the client device and responsive to the NL base input. In additional or alternative implementations, the NL based input can be obtained from database(s). In these implementations, a given candidate response that is progressively refined can be utilized in fine-tuning of the LLM.