公开(公告)号：US11487939B2

公开(公告)日：2022-11-01

申请号：US16549985

申请日：2019-08-23

Applicant: salesforce.com, inc.

Inventor： Tong Niu ,  Caiming Xiong ,  Richard Socher

IPC: G06F40/284 ,  G06N3/08 ,  H03M7/42 ,  H03M7/30 ,  G06F40/40

Abstract: Embodiments described herein provide a provide a fully unsupervised model for text compression. Specifically, the unsupervised model is configured to identify an optimal deletion path for each input sequence of texts (e.g., a sentence) and words from the input sequence are gradually deleted along the deletion path. To identify the optimal deletion path, the unsupervised model may adopt a pretrained bidirectional language model (BERT) to score each candidate deletion based on the average perplexity of the resulting sentence and performs a simple greedy look-ahead tree search to select the best deletion for each step.

公开(公告)号：US11409945B2

公开(公告)日：2022-08-09

申请号：US17027130

申请日：2020-09-21

Applicant: salesforce.com, inc.

Inventor： Bryan McCann ,  Caiming Xiong ,  Richard Socher

IPC: G06F40/126 ,  G06N3/08 ,  G06N3/04 ,  G06F40/30 ,  G06F40/47 ,  G06F40/205 ,  G06F40/289 ,  G06F40/44 ,  G06F40/58

Abstract: A system is provided for natural language processing. In some embodiments, the system includes an encoder for generating context-specific word vectors for at least one input sequence of words. The encoder is pre-trained using training data for performing a first natural language processing task. A neural network performs a second natural language processing task on the at least one input sequence of words using the context-specific word vectors. The first natural language process task is different from the second natural language processing task and the neural network is separately trained from the encoder. In some embodiments, the first natural processing task can be machine translation, and the second natural processing task can be one of sentiment analysis, question classification, entailment classification, and question answering.

公开(公告)号：US11276002B2

公开(公告)日：2022-03-15

申请号：US15926768

申请日：2018-03-20

Applicant: salesforce.com, inc.

Inventor： Nitish Shirish Keskar ,  Richard Socher

IPC: G06N3/08 ,  G06N3/04

Abstract: Hybrid training of deep networks includes a multi-layer neural network. The training includes setting a current learning algorithm for the multi-layer neural network to a first learning algorithm. The training further includes iteratively applying training data to the neural network, determining a gradient for parameters of the neural network based on the applying of the training data, updating the parameters based on the current learning algorithm, and determining whether the current learning algorithm should be switched to a second learning algorithm based on the updating. The training further includes, in response to the determining that the current learning algorithm should be switched to a second learning algorithm, changing the current learning algorithm to the second learning algorithm and initializing a learning rate of the second learning algorithm based on the gradient and a step used by the first learning algorithm to update the parameters of the neural network.

公开(公告)号：US20220067277A1

公开(公告)日：2022-03-03

申请号：US17002562

申请日：2020-08-25

Applicant: salesforce.com, inc.

Inventor： Shiva Kumar Pentyala ,  Mridul Gupta ,  Ankit Chadha ,  Indira Iyer ,  Richard Socher

IPC: G06F40/253 ,  G06F40/30 ,  G10L15/19

Abstract: A natural language processing system that trains task models for particular natural language tasks programmatically generates additional utterances for inclusion in the training set, based on the existing utterances in the training set and the existing state of a task model as generated from the original (non-augmented) training set. More specifically, the training augmentation module 220 identifies specific textual units of utterances and generates variants of the utterances based on those identified units. The identification is based on determined importances of the textual units to the output of the task model, as well as on task rules that correspond to the natural language task for which the task model is being generated. The generation of the additional utterances improves the quality of the task model without the expense of manual labeling of utterances for training set inclusion.

公开(公告)号：US11222253B2

公开(公告)日：2022-01-11

申请号：US15421424

申请日：2017-01-31

Applicant: salesforce.com, inc.

Inventor： Kazuma Hashimoto ,  Caiming Xiong ,  Richard Socher

IPC: G06N3/04 ,  G06N3/08 ,  G06F40/30 ,  G06F40/205 ,  G06F40/216 ,  G06F40/253 ,  G06F40/284 ,  G06N3/063 ,  G10L15/18 ,  G10L25/30 ,  G10L15/16 ,  G06F40/00

Abstract: The technology disclosed provides a so-called “joint many-task neural network model” to solve a variety of increasingly complex natural language processing (NLP) tasks using growing depth of layers in a single end-to-end model. The model is successively trained by considering linguistic hierarchies, directly connecting word representations to all model layers, explicitly using predictions in lower tasks, and applying a so-called “successive regularization” technique to prevent catastrophic forgetting. Three examples of lower level model layers are part-of-speech (POS) tagging layer, chunking layer, and dependency parsing layer. Two examples of higher level model layers are semantic relatedness layer and textual entailment layer. The model achieves the state-of-the-art results on chunking, dependency parsing, semantic relatedness and textual entailment.

公开(公告)号：US20210374353A1

公开(公告)日：2021-12-02

申请号：US17005316

申请日：2020-08-28

Applicant: salesforce.com, inc.

Inventor： Jianguo Zhang ,  Kazuma Hashimoto ,  Chien-Sheng Wu ,  Wenhao Liu ,  Richard Socher ,  Caiming Xiong

IPC: G06F40/30 ,  G10L15/16

Abstract: An online system allows user interactions using natural language expressions. The online system uses a machine learning based model to infer an intent represented by a user expression. The machine learning based model takes as input a user expression and an example expression to compute a score indicating whether the user expression matches the example expression. Based on the scores, the intent inference module determines a most applicable intent for the expression. The online system determines a confidence threshold such that user expressions indicating a high confidence are assigned the most applicable intent and user expressions indicating a low confidence are assigned an out-of-scope intent. The online system encodes the example expressions using the machine learning based model. The online system may compare an encoded user expression with encoded example expressions to identify a subset of example expressions used to determine the most applicable intent.

公开(公告)号：US11170287B2

公开(公告)日：2021-11-09

申请号：US15881582

申请日：2018-01-26

Applicant: salesforce.com, inc.

Inventor： Victor Zhong ,  Caiming Xiong ,  Richard Socher

IPC: G06N3/04 ,  G06N3/08 ,  G06N5/04 ,  G06F16/93 ,  G06F16/2458 ,  G06N3/00

Abstract: A computer-implemented method for dual sequence inference using a neural network model includes generating a codependent representation based on a first input representation of a first sequence and a second input representation of a second sequence using an encoder of the neural network model and generating an inference based on the codependent representation using a decoder of the neural network model. The neural network model includes a plurality of model parameters learned according to a machine learning process. The encoder includes a plurality of coattention layers arranged sequentially, each coattention layer being configured to receive a pair of layer input representations and generate one or more summary representations, and an output layer configured to receive the one or more summary representations from a last layer among the plurality of coattention layers and generate the codependent representation.

公开(公告)号：US10963652B2

公开(公告)日：2021-03-30

申请号：US16264392

申请日：2019-01-31

Applicant: salesforce.com, inc.

Inventor： Kazuma Hashimoto ,  Raffaella Buschiazzo ,  James Bradbury ,  Teresa Marshall ,  Caiming Xiong ,  Richard Socher

IPC: G06F40/58 ,  G06N3/08

Abstract: Approaches for the translation of structured text include an embedding module for encoding and embedding source text in a first language, an encoder for encoding output of the embedding module, a decoder for iteratively decoding output of the encoder based on generated tokens in translated text from previous iterations, a beam module for constraining output of the decoder with respect to possible embedded tags to include in the translated text for a current iteration using a beam search, and a layer for selecting a token to be included in the translated text for the current iteration. The translated text is in a second language different from the first language. In some embodiments, the approach further includes scoring and pointer modules for selecting the token based on the output of the beam module or copied from the source text or reference text from a training pair best matching the source text.

公开(公告)号：US10817650B2

公开(公告)日：2020-10-27

申请号：US15982841

申请日：2018-05-17

Applicant: salesforce.com, inc.

Inventor： Bryan McCann ,  Caiming Xiong ,  Richard Socher

IPC: G06F40/126 ,  G06N3/08 ,  G06N3/04 ,  G06F40/30 ,  G06F40/47 ,  G06F40/205 ,  G06F40/289 ,  G06F40/44 ,  G06F40/58

Abstract: A system is provided for natural language processing. In some embodiments, the system includes an encoder for generating context-specific word vectors for at least one input sequence of words. The encoder is pre-trained using training data for performing a first natural language processing task. A neural network performs a second natural language processing task on the at least one input sequence of words using the context-specific word vectors. The first natural language process task is different from the second natural language processing task and the neural network is separately trained from the encoder. In some embodiments, the first natural processing task can be machine translation, and the second natural processing task can be one of sentiment analysis, question classification, entailment classification, and question answering.

公开(公告)号：US20200334334A1

公开(公告)日：2020-10-22

申请号：US16518905

申请日：2019-07-22

Applicant: salesforce.com, inc.

Inventor： Nitish Shirish Keskar ,  Bryan McCann ,  Richard Socher ,  Caiming Xiong

IPC: G06F17/27

Abstract: Systems and methods for unifying question answering and text classification via span extraction include a preprocessor for preparing a source text and an auxiliary text based on a task type of a natural language processing task, an encoder for receiving the source text and the auxiliary text from the preprocessor and generating an encoded representation of a combination of the source text and the auxiliary text, and a span-extractive decoder for receiving the encoded representation and identifying a span of text within the source text that is a result of the NLP task. The task type is one of entailment, classification, or regression. In some embodiments, the source text includes one or more of text received as input when the task type is entailment, a list of classifications when the task type is entailment or classification, or a list of similarity options when the task type is regression.