公开(公告)号：US20230342552A1

公开(公告)日：2023-10-26

申请号：US17889178

申请日：2022-08-16

Applicant: Salesforce, Inc.

Inventor： Rishabh Bhardwaj ,  Amrita Saha ,  Chu Hong Hoi

IPC: G06F40/284 ,  G06F40/289 ,  G06N20/00

CPC classification number: G06F40/284 ,  G06F40/289 ,  G06N20/00

Abstract: Embodiments described herein provide a soft prompt tuning technique referred to as the Vector quantized Input-contextualized Prompt (VIP). The VIP techniques has two integral properties i) instead of learning a fixed set of prompt tokens irrespective of the input, it generates a contextualized version of the soft prompts, conditional on the input text ii) it further passes the input-contextualized prompt tokens through a quantization network, inspired by Vector Quantized Transformers. The quantization network uses nearest neighbor search over a learnable codebook to train a discrete latent variable model over the prompt-space, thus generating quantized version of contextual prompt tokens. These quantized contextual prompt tokens are finally fed into the frozen language model along with the original input text.

公开(公告)号：US12147765B2

公开(公告)日：2024-11-19

申请号：US17889178

申请日：2022-08-16

Applicant: Salesforce, Inc.

Inventor： Rishabh Bhardwaj ,  Amrita Saha ,  Chu Hong Hoi

IPC: G06F40/284 ,  G06F40/12 ,  G06F40/289 ,  G06F40/40 ,  G06N20/00

Abstract: Embodiments described herein provide a soft prompt tuning technique referred to as the Vector quantized Input-contextualized Prompt (VIP). The VIP techniques has two integral properties i) instead of learning a fixed set of prompt tokens irrespective of the input, it generates a contextualized version of the soft prompts, conditional on the input text ii) it further passes the input-contextualized prompt tokens through a quantization network, inspired by Vector Quantized Transformers. The quantization network uses nearest neighbor search over a learnable codebook to train a discrete latent variable model over the prompt-space, thus generating quantized version of contextual prompt tokens. These quantized contextual prompt tokens are finally fed into the frozen language model along with the original input text.

公开(公告)号：US20230342559A1

公开(公告)日：2023-10-26

申请号：US17889175

申请日：2022-08-16

Applicant: Salesforce, Inc.

Inventor： Rishabh Bhardwaj ,  Amrita Saha ,  Chu Hong Hoi

IPC: G06F40/40 ,  G06F40/284 ,  G06F40/12

CPC classification number: G06F40/40 ,  G06F40/284 ,  G06F40/12

Abstract: Embodiments described herein provide a soft prompt tuning technique referred to as the Vector quantized Input-contextualized Prompt (VIP). The VIP techniques has two integral properties i) instead of learning a fixed set of prompt tokens irrespective of the input, it generates a contextualized version of the soft prompts, conditional on the input text ii) it further passes the input-contextualized prompt tokens through a quantization network, inspired by Vector Quantized Transformers. The quantization network uses nearest neighbor search over a learnable codebook to train a discrete latent variable model over the prompt-space, thus generating quantized version of contextual prompt tokens. These quantized contextual prompt tokens are finally fed into the frozen language model along with the original input text.