公开(公告)号：US20230073754A1

公开(公告)日：2023-03-09

申请号：US17586451

申请日：2022-01-27

Applicant: salesforce.com, inc.

Inventor： Yongjun Chen ,  Zhiwei Liu ,  Jia Li ,  Caiming Xiong

IPC: G06K9/62

Abstract: Embodiments described herein provides an intent prototypical contrastive learning framework that leverages intent similarities between users with different behavior sequences. Specifically, user behavior sequences are encoded into a plurality of user interest representations. The user interest representations are clustered into a plurality of clusters based on mutual distances among the user interest representations in a representation space. Intention prototypes are determined based on centroids of the clusters. A set of augmented views for user behavior sequences are created and encoded into a set of view representations. A contrastive loss is determined based on the set of augmented views and the plurality of intention prototypes. Model parameters are updated based at least in part on the contrastive loss.

公开(公告)号：US11580977B2

公开(公告)日：2023-02-14

申请号：US17037556

申请日：2020-09-29

Applicant: salesforce.com, inc.

Inventor： Xinyi Yang ,  Tian Xie ,  Caiming Xiong ,  Wenhao Liu ,  Huan Wang ,  Kazuma Hashimoto ,  Yingbo Zhou ,  Xugang Ye ,  Jin Qu ,  Feihong Wu

IPC: G10L15/22 ,  G10L15/16 ,  G10L15/26 ,  G10L15/30

Abstract: A conversation engine performs conversations with users using chatbots customized for performing a set of tasks that can be performed using an online system. The conversation engine loads a chatbot configuration that specifies the behavior of a chatbot including the tasks that can be performed by the chatbot, the types of entities relevant to each task, and so on. The conversation may be voice based and use natural language. The conversation engine may load different chatbot configurations to implement different chatbots. The conversation engine receives a conversation engine configuration that specifies the behavior of the conversation engine across chatbots. The system may be a multi-tenant system that allows customization of the chatbots for each tenant.

公开(公告)号：US11580359B2

公开(公告)日：2023-02-14

申请号：US16664508

申请日：2019-10-25

Applicant: salesforce.com, inc.

Inventor： Stephen Joseph Merity ,  Caiming Xiong ,  James Bradbury ,  Richard Socher

IPC: G06N3/04 ,  G06N3/084 ,  G06F40/284 ,  G06N3/08 ,  G06N7/00

Abstract: The technology disclosed provides a so-called “pointer sentinel mixture architecture” for neural network sequence models that has the ability to either reproduce a token from a recent context or produce a token from a predefined vocabulary. In one implementation, a pointer sentinel-LSTM architecture achieves state of the art language modeling performance of 70.9 perplexity on the Penn Treebank dataset, while using far fewer parameters than a standard softmax LSTM.

公开(公告)号：US11562142B2

公开(公告)日：2023-01-24

申请号：US17187608

申请日：2021-02-26

Applicant: salesforce.com, inc.

Inventor： Erik Lennart Nijkamp ,  Caiming Xiong

IPC: G06F40/279 ,  G06F40/58

Abstract: A machine learning based model generates a feature representation of a text sequence, for example, a natural language sentence or phrase. The system trains the machine learning based model by receiving an input text sequence and perturbing the input text sequence by masking a subset of tokens. The machine learning based model is used to predict the masked tokens. A predicted text sequence is generated based on the predictions of the masked tokens. The system processes the predicted text sequence using the machine learning based model to determine whether a token was predicted or an original token. The parameters of the machine learning based model are adjusted to minimize an aggregate loss based on prediction of the correct word for a masked token and a classification of a word as original or replaced.

公开(公告)号：US11531821B2

公开(公告)日：2022-12-20

申请号：US16993257

申请日：2020-08-13

Applicant: salesforce.com, inc.

Inventor： Tian Xie ,  Xinyi Yang ,  Caiming Xiong ,  Wenhao Liu ,  Huan Wang ,  Wenpeng Yin ,  Jin Qu

IPC: G06F40/30 ,  G10L15/22 ,  H04L51/02

Abstract: A system performs conversations with users using chatbots customized for performing a set of tasks. The system may be a multi-tenant system that allows customization of the chatbots for each tenant. The system processes sentences that may include negation or coreferences. The system determines a confidence score for an input sentence using an intent detection model, for example, a neural network. The system modifies the sentence to generate a modified sentence, for example, by removing a negation or by replacing a pronoun with an entity. The system generates a confidence score for the modified sentence using the intent detection model. The system determines the intent of the sentence based on the confidence scores of the sentence and the modified sentence. The system performs tasks based on the determined intent and performs conversations with users based on the tasks.

公开(公告)号：US11487939B2

公开(公告)日：2022-11-01

申请号：US16549985

申请日：2019-08-23

Applicant: salesforce.com, inc.

Inventor： Tong Niu ,  Caiming Xiong ,  Richard Socher

IPC: G06F40/284 ,  G06N3/08 ,  H03M7/42 ,  H03M7/30 ,  G06F40/40

Abstract: Embodiments described herein provide a provide a fully unsupervised model for text compression. Specifically, the unsupervised model is configured to identify an optimal deletion path for each input sequence of texts (e.g., a sentence) and words from the input sequence are gradually deleted along the deletion path. To identify the optimal deletion path, the unsupervised model may adopt a pretrained bidirectional language model (BERT) to score each candidate deletion based on the average perplexity of the resulting sentence and performs a simple greedy look-ahead tree search to select the best deletion for each step.

公开(公告)号：US11481636B2

公开(公告)日：2022-10-25

申请号：US16877325

申请日：2020-05-18

Applicant: salesforce.com, inc.

Inventor： Govardana Sachithanandam Ramachandran ,  Ka Chun Au ,  Shashank Harinath ,  Wenhao Liu ,  Alexis Roos ,  Caiming Xiong

IPC: G06N3/08 ,  G06N20/00 ,  G06K9/62 ,  G06N20/10 ,  G06F17/18 ,  G06V10/75

Abstract: An embodiment provided herein preprocesses the input samples to the classification neural network, e.g., by adding Gaussian noise to word/sentence representations to make the function of the neural network satisfy Lipschitz property such that a small change in the input does not cause much change to the output if the input sample is in-distribution. Method to induce properties in the feature representation of neural network such that for out-of-distribution examples the feature representation magnitude is either close to zero or the feature representation is orthogonal to all class representations. Method to generate examples that are structurally similar to in-domain and semantically out-of domain for use in out-of-domain classification training. Method to prune feature representation dimension to mitigate long tail error of unused dimension in out-of-domain classification. Using these techniques, the accuracy of both in-domain and out-of-distribution identification can be improved.

公开(公告)号：US20220277141A1

公开(公告)日：2022-09-01

申请号：US17187608

申请日：2021-02-26

Applicant: salesforce.com, inc.

Inventor： Erik Lennart Nijkamp ,  Caiming Xiong

IPC: G06F40/279 ,  G06F40/58

Abstract: A machine learning based model generates a feature representation of a text sequence, for example, a natural language sentence or phrase. The system trains the machine learning based model by receiving an input text sequence and perturbing the input text sequence by masking a subset of tokens. The machine learning based model is used to predict the masked tokens. A predicted text sequence is generated based on the predictions of the masked tokens. The system processes the predicted text sequence using the machine learning based model to determine whether a token was predicted or an original token. The parameters of the machine learning based model are adjusted to minimize an aggregate loss based on prediction of the correct word for a masked token and a classification of a word as original or replaced.

公开(公告)号：US11409945B2

公开(公告)日：2022-08-09

申请号：US17027130

申请日：2020-09-21

Applicant: salesforce.com, inc.

Inventor： Bryan McCann ,  Caiming Xiong ,  Richard Socher

IPC: G06F40/126 ,  G06N3/08 ,  G06N3/04 ,  G06F40/30 ,  G06F40/47 ,  G06F40/205 ,  G06F40/289 ,  G06F40/44 ,  G06F40/58

Abstract: A system is provided for natural language processing. In some embodiments, the system includes an encoder for generating context-specific word vectors for at least one input sequence of words. The encoder is pre-trained using training data for performing a first natural language processing task. A neural network performs a second natural language processing task on the at least one input sequence of words using the context-specific word vectors. The first natural language process task is different from the second natural language processing task and the neural network is separately trained from the encoder. In some embodiments, the first natural processing task can be machine translation, and the second natural processing task can be one of sentiment analysis, question classification, entailment classification, and question answering.

公开(公告)号：US11386327B2

公开(公告)日：2022-07-12

申请号：US15983782

申请日：2018-05-18

Applicant: salesforce.com, inc.

Inventor： Huishuai Zhang ,  Caiming Xiong

IPC: G06N3/08 ,  G06N3/04 ,  G10L15/16 ,  G06F17/16 ,  G06F17/11 ,  G06N7/00 ,  G06N5/00

Abstract: Embodiments for training a neural network are provided. A neural network is divided into a first block and a second block, and the parameters in the first block and second block are trained in parallel. To train the parameters, a gradient from a gradient mini-batch included in training data is generated. A curvature-vector product from a curvature mini-batch included in the training data is also generated. The gradient and the curvature-vector product generate a conjugate gradient. The conjugate gradient is used to determine a change in parameters in the first block in parallel with a change in parameters in the second block. The curvature matrix in the curvature-vector product includes zero values when the terms correspond to parameters from different blocks.