公开(公告)号：US20240412059A1

公开(公告)日：2024-12-12

申请号：US18330488

申请日：2023-06-07

Applicant: Salesforce, Inc.

Inventor： Regunathan Radhakrishnan ,  Zachary Alexander ,  Sitaram Asur ,  Shashank Harinath ,  Na Cheng ,  Shiva Kumar Pentyala

IPC: G06N3/08

Abstract: Embodiments described herein provide A method for training a neural network based model. The methods include receiving a training dataset with a plurality of training samples, and those samples are encoded into representations in feature space. A positive sample is determined from the raining dataset based on a relationship between the given query and the positive sample in feature space. For a given query, a positive sample from the training dataset is selected based on a relationship between the given query and the positive sample in a feature space. One or more negative samples from the training dataset that are within a reconfigurable distance to the positive sample in the feature space are selected, and a loss is computed based on the positive sample and the one or more negative samples. The neural network is trained based on the loss.

公开(公告)号：US12197317B2

公开(公告)日：2025-01-14

申请号：US18156323

申请日：2023-01-18

Applicant: Salesforce, Inc.

Inventor： Shiva Kumar Pentyala ,  Shashank Harinath ,  Sitaram Asur ,  Zachary Alexander

IPC: G06F11/36

Abstract: Embodiments described herein provide an automated testing pipeline for providing a testing dataset for testing a trained neural network model trained using a first training dataset. A first testing dataset for the trained neural network including a first plurality of user queries is received. A dependency parser is used to filter the first plurality of user queries based on one or more action verbs. A pretrained language model is used to rank the remaining user queries based on respective relationships with queries in the first training dataset. Further, user queries that are classified as keyword matches with the queries in the first training dataset using a bag of words classifier are removed. A second testing dataset is generated using the ranked remaining user queries. Testing outputs are generated, by the trained neural network model, using the second testing dataset.

公开(公告)号：US12019984B2

公开(公告)日：2024-06-25

申请号：US17479748

申请日：2021-09-20

Applicant: Salesforce, Inc.

Inventor： Shilpa Bhagavath ,  Shubham Mehrotra ,  Abhishek Sharma ,  Shashank Harinath ,  Na Cheng ,  Zineb Laraki

IPC: G06F40/263 ,  G06F18/2415 ,  G06F18/2431 ,  G06F40/35 ,  G06F40/58 ,  G10L15/22

CPC classification number: G06F40/263 ,  G06F18/2415 ,  G06F18/2431 ,  G06F40/35 ,  G06F40/58 ,  G10L15/22

Abstract: A method that includes receiving an input at an interactive conversation service that uses an intent classification model. The method may further include generating, using an encoder model of the intent classification model, a set of output vectors corresponding to the input, where the encoder model is configured to determine a set of metrics corresponding to intent classifications. The method may further include determining, using an outlier detection model of the intent classification model, whether the input is in-domain or out-of-domain (OOD) based on a first vector of the set of output vectors satisfying a domain threshold relative to one or more of the intent classifications. The method may further include outputting, by the intent classification model, a second vector of the set of output vectors that indicates the set of metrics corresponding to the intent classifications or an indication that the input is OOD.

公开(公告)号：US20240411992A1

公开(公告)日：2024-12-12

申请号：US18335898

申请日：2023-06-15

Applicant: Salesforce, Inc.

Inventor： Shiva Kumar Pentyala ,  Prafulla Kumar Choubey ,  Shashank Harinath ,  Sitaram Asur ,  Chien-Sheng Jason Wu ,  Zachary Alexander ,  Caiming Xiong

IPC: G06F40/284 ,  G06N3/08

Abstract: Embodiments described herein provide a training framework for generative NLP models. Specifically, the training input, e.g., in the form of a sequence of tokens representing a user-agent dialogue, may be randomly masked for a few spans, which can be one or more tokens, one or more words, one or more sentences, or one or more paragraphs. These masked spans are replaced with their embeddings generated from pre-trained large language models are then used for training the NLP model.

公开(公告)号：US20240411991A1

公开(公告)日：2024-12-12

申请号：US18330216

申请日：2023-06-06

Applicant: Salesforce, Inc.

Inventor： Shiva Kumar Pentyala ,  Prafulla Kumar Choubey ,  Shashank Harinath ,  Sitaram Asur ,  Chien-Sheng Jason Wu ,  Zachary Alexander ,  Caiming Xiong

IPC: G06F40/284

Abstract: Embodiments described herein provide a training framework for generative NLP models that operate on previously learnt knowledge from pretrained large language models. Specifically, to train an NLP model to generate a response to a user utterance (e.g., “resolve login issue”), document embeddings of support IT documents encoded by a pretrained LLM are fed to an NLP decoder together with a training dialogue (e.g., a dialogue between the chat agent on how to “resolve login issue”). The NLP decoder can thus be trained by a causal language modeling loss computed based on the predicted next token and the ground-truth token from the training dialogue.

公开(公告)号：US20240303473A1

公开(公告)日：2024-09-12

申请号：US18496513

申请日：2023-10-27

Applicant: Salesforce, Inc.

Inventor： Na (Claire) Cheng ,  Jayesh Govindarajan ,  Zachary Alexander ,  Shashank Harinath ,  Atul Kshirsagar ,  Fermin Ordaz

IPC: G06N3/0475

CPC classification number: G06N3/0475

Abstract: Embodiments provide a generative AI creation framework to a customized generative AI stack using a foundational model (such as GPT) based on user-defined prompts, a natural language description of the task to be accomplished, and domain adaptation. In one embodiment, organization-specific knowledge may be injected into either the prompt and/or the foundational model. In this way, the customized generative AI stack thus supports a full spectrum of domain-adaptive prompts to enable a full spectrum of personalized and adaptive AI chat applications.

公开(公告)号：US11880659B2

公开(公告)日：2024-01-23

申请号：US17162318

申请日：2021-01-29

Applicant: Salesforce, Inc.

Inventor： Shiva Kumar Pentyala ,  Jean-Marc Soumet ,  Shashank Harinath ,  Shilpa Bhagavath ,  Johnson Liu ,  Ankit Chadha

IPC: G06F17/00 ,  G06F40/30 ,  G06F40/295 ,  G06N20/00

CPC classification number: G06F40/30 ,  G06F40/295 ,  G06N20/00

Abstract: Methods and systems for hierarchical natural language understanding are described. A representation of an utterance is inputted to a first machine learning model to obtain information on the first utterance. According to the information on the utterance a determination that the representation of the utterance is to be inputted to a second machine learning model that performs a dedicated natural language task is performed. In response to determining that the representation of the utterance is to be inputted to a second machine learning model, the utterance is inputted to the second machine learning model to obtain an output of the dedicated natural language task.

公开(公告)号：US20240303443A1

公开(公告)日：2024-09-12

申请号：US18496523

申请日：2023-10-27

Applicant: Salesforce, Inc.

Inventor： Na (Claire) Cheng ,  Jayesh Govindarajan ,  Zachary Alexander ,  Shashank Harinath ,  Atul Kshirsagar ,  Fermin Ordaz

IPC: G06F40/40 ,  G06F16/33 ,  G06F40/295

CPC classification number: G06F40/40 ,  G06F16/3347 ,  G06F40/295

Abstract: Embodiments provide a generative AI creation framework to a customized generative AI stack using a foundational model (such as GPT) based on user-defined prompts, a natural language description of the task to be accomplished, and domain adaptation. In one embodiment, organization-specific knowledge may be injected into either the prompt and/or the foundational model. In this way, the customized generative AI stack thus supports a full spectrum of domain-adaptive prompts to enable a full spectrum of personalized and adaptive AI chat applications.

公开(公告)号：US20240242022A1

公开(公告)日：2024-07-18

申请号：US18156043

申请日：2023-01-18

Applicant: Salesforce, Inc.

Inventor： Victor Yee ,  Chien-Sheng Wu ,  Na Cheng ,  Alexander R. Fabbri ,  Zachary Alexander ,  Nicholas Feinig ,  Sameer Abhinkar ,  Shashank Harinath ,  Sitaram Asur ,  Jacob Nathaniel Huffman ,  Wojciech Kryscinski ,  Caiming Xiong

IPC: G06F40/174 ,  G06F16/34

CPC classification number: G06F40/174 ,  G06F16/345

Abstract: Embodiments described herein provide a structured conversation summarization framework. A user interface may be provided which allows an agent to perform a conversation with a customer, for example regarding resolving a customer support issue. Utterances by both the agent and customer may be stored, and at the end of the conversation, the utterances may be used to generate a structured summary. The structured summary may include components such as a general summary, an issue summary, and a resolution summary. Using neural network models and heuristics, each component of the summary may be automatically generated.

公开(公告)号：US20240143945A1

公开(公告)日：2024-05-02

申请号：US18161767

申请日：2023-01-30

Applicant: Salesforce, Inc.

Inventor： Shubham Mehrotra ,  Zachary Alexander ,  Shilpa Bhagavath ,  Gurkirat Singh ,  Shashank Harinath ,  Anuprit Kale

IPC: G06F40/47 ,  G06F40/30 ,  G06F40/51

CPC classification number: G06F40/47 ,  G06F40/30 ,  G06F40/51

Abstract: Embodiments described herein provide a cross-lingual intent classification model that predicts in multiple languages without the need of training data in all the multiple languages. For example, data requirement for training can be reduced to just one utterance per intent label. Specifically, when an utterance is fed to the intent classification model, the model checks whether the utterance is similar to any of the example utterances provided for each intent. If any such utterance(s) are found, the model returns the specified intent, otherwise, it returns out of domain (OOD).