公开(公告)号：US11276002B2

公开(公告)日：2022-03-15

申请号：US15926768

申请日：2018-03-20

申请人： salesforce.com, inc.

发明人： Nitish Shirish Keskar ,  Richard Socher

IPC分类号： G06N3/08 ,  G06N3/04

摘要： Hybrid training of deep networks includes a multi-layer neural network. The training includes setting a current learning algorithm for the multi-layer neural network to a first learning algorithm. The training further includes iteratively applying training data to the neural network, determining a gradient for parameters of the neural network based on the applying of the training data, updating the parameters based on the current learning algorithm, and determining whether the current learning algorithm should be switched to a second learning algorithm based on the updating. The training further includes, in response to the determining that the current learning algorithm should be switched to a second learning algorithm, changing the current learning algorithm to the second learning algorithm and initializing a learning rate of the second learning algorithm based on the gradient and a step used by the first learning algorithm to update the parameters of the neural network.

公开(公告)号：US20220067277A1

公开(公告)日：2022-03-03

申请号：US17002562

申请日：2020-08-25

申请人： salesforce.com, inc.

发明人： Shiva Kumar Pentyala ,  Mridul Gupta ,  Ankit Chadha ,  Indira Iyer ,  Richard Socher

IPC分类号： G06F40/253 ,  G06F40/30 ,  G10L15/19

摘要： A natural language processing system that trains task models for particular natural language tasks programmatically generates additional utterances for inclusion in the training set, based on the existing utterances in the training set and the existing state of a task model as generated from the original (non-augmented) training set. More specifically, the training augmentation module 220 identifies specific textual units of utterances and generates variants of the utterances based on those identified units. The identification is based on determined importances of the textual units to the output of the task model, as well as on task rules that correspond to the natural language task for which the task model is being generated. The generation of the additional utterances improves the quality of the task model without the expense of manual labeling of utterances for training set inclusion.

公开(公告)号：US11222253B2

公开(公告)日：2022-01-11

申请号：US15421424

申请日：2017-01-31

申请人： salesforce.com, inc.

发明人： Kazuma Hashimoto ,  Caiming Xiong ,  Richard Socher

IPC分类号： G06N3/04 ,  G06N3/08 ,  G06F40/30 ,  G06F40/205 ,  G06F40/216 ,  G06F40/253 ,  G06F40/284 ,  G06N3/063 ,  G10L15/18 ,  G10L25/30 ,  G10L15/16 ,  G06F40/00

摘要： The technology disclosed provides a so-called “joint many-task neural network model” to solve a variety of increasingly complex natural language processing (NLP) tasks using growing depth of layers in a single end-to-end model. The model is successively trained by considering linguistic hierarchies, directly connecting word representations to all model layers, explicitly using predictions in lower tasks, and applying a so-called “successive regularization” technique to prevent catastrophic forgetting. Three examples of lower level model layers are part-of-speech (POS) tagging layer, chunking layer, and dependency parsing layer. Two examples of higher level model layers are semantic relatedness layer and textual entailment layer. The model achieves the state-of-the-art results on chunking, dependency parsing, semantic relatedness and textual entailment.

公开(公告)号：US20210374353A1

公开(公告)日：2021-12-02

申请号：US17005316

申请日：2020-08-28

申请人： salesforce.com, inc.

发明人： Jianguo Zhang ,  Kazuma Hashimoto ,  Chien-Sheng Wu ,  Wenhao Liu ,  Richard Socher ,  Caiming Xiong

IPC分类号： G06F40/30 ,  G10L15/16

摘要： An online system allows user interactions using natural language expressions. The online system uses a machine learning based model to infer an intent represented by a user expression. The machine learning based model takes as input a user expression and an example expression to compute a score indicating whether the user expression matches the example expression. Based on the scores, the intent inference module determines a most applicable intent for the expression. The online system determines a confidence threshold such that user expressions indicating a high confidence are assigned the most applicable intent and user expressions indicating a low confidence are assigned an out-of-scope intent. The online system encodes the example expressions using the machine learning based model. The online system may compare an encoded user expression with encoded example expressions to identify a subset of example expressions used to determine the most applicable intent.

公开(公告)号：US11170287B2

公开(公告)日：2021-11-09

申请号：US15881582

申请日：2018-01-26

申请人： salesforce.com, inc.

发明人： Victor Zhong ,  Caiming Xiong ,  Richard Socher

IPC分类号： G06N3/04 ,  G06N3/08 ,  G06N5/04 ,  G06F16/93 ,  G06F16/2458 ,  G06N3/00

摘要： A computer-implemented method for dual sequence inference using a neural network model includes generating a codependent representation based on a first input representation of a first sequence and a second input representation of a second sequence using an encoder of the neural network model and generating an inference based on the codependent representation using a decoder of the neural network model. The neural network model includes a plurality of model parameters learned according to a machine learning process. The encoder includes a plurality of coattention layers arranged sequentially, each coattention layer being configured to receive a pair of layer input representations and generate one or more summary representations, and an output layer configured to receive the one or more summary representations from a last layer among the plurality of coattention layers and generate the codependent representation.

公开(公告)号：US10963652B2

公开(公告)日：2021-03-30

申请号：US16264392

申请日：2019-01-31

申请人： salesforce.com, inc.

发明人： Kazuma Hashimoto ,  Raffaella Buschiazzo ,  James Bradbury ,  Teresa Marshall ,  Caiming Xiong ,  Richard Socher

IPC分类号： G06F40/58 ,  G06N3/08

摘要： Approaches for the translation of structured text include an embedding module for encoding and embedding source text in a first language, an encoder for encoding output of the embedding module, a decoder for iteratively decoding output of the encoder based on generated tokens in translated text from previous iterations, a beam module for constraining output of the decoder with respect to possible embedded tags to include in the translated text for a current iteration using a beam search, and a layer for selecting a token to be included in the translated text for the current iteration. The translated text is in a second language different from the first language. In some embodiments, the approach further includes scoring and pointer modules for selecting the token based on the output of the beam module or copied from the source text or reference text from a training pair best matching the source text.

公开(公告)号：US10817650B2

公开(公告)日：2020-10-27

申请号：US15982841

申请日：2018-05-17

申请人： salesforce.com, inc.

发明人： Bryan McCann ,  Caiming Xiong ,  Richard Socher

IPC分类号： G06F40/126 ,  G06N3/08 ,  G06N3/04 ,  G06F40/30 ,  G06F40/47 ,  G06F40/205 ,  G06F40/289 ,  G06F40/44 ,  G06F40/58

摘要： A system is provided for natural language processing. In some embodiments, the system includes an encoder for generating context-specific word vectors for at least one input sequence of words. The encoder is pre-trained using training data for performing a first natural language processing task. A neural network performs a second natural language processing task on the at least one input sequence of words using the context-specific word vectors. The first natural language process task is different from the second natural language processing task and the neural network is separately trained from the encoder. In some embodiments, the first natural processing task can be machine translation, and the second natural processing task can be one of sentiment analysis, question classification, entailment classification, and question answering.

公开(公告)号：US20200334334A1

公开(公告)日：2020-10-22

申请号：US16518905

申请日：2019-07-22

申请人： salesforce.com, inc.

发明人： Nitish Shirish Keskar ,  Bryan McCann ,  Richard Socher ,  Caiming Xiong

IPC分类号： G06F17/27

摘要： Systems and methods for unifying question answering and text classification via span extraction include a preprocessor for preparing a source text and an auxiliary text based on a task type of a natural language processing task, an encoder for receiving the source text and the auxiliary text from the preprocessor and generating an encoded representation of a combination of the source text and the auxiliary text, and a span-extractive decoder for receiving the encoded representation and identifying a span of text within the source text that is a result of the NLP task. The task type is one of entailment, classification, or regression. In some embodiments, the source text includes one or more of text received as input when the task type is entailment, a list of classifications when the task type is entailment or classification, or a list of similarity options when the task type is regression.

公开(公告)号：US20200302178A1

公开(公告)日：2020-09-24

申请号：US16394964

申请日：2019-04-25

申请人： salesforce.com, inc.

发明人： Mingfei Gao ,  Richard Socher ,  Caiming Xiong

IPC分类号： G06K9/00 ,  G06K9/62 ,  G06N3/04

摘要： Embodiments described herein provide a two-stage online detection of action start system including a classification module and a localization module. The classification module generates a set of action scores corresponding to a first video frame from the video, based on the first video frame and video frames before the first video frames in the video. Each action score indicating a respective probability that the first video frame contains a respective action class. The localization module is coupled to the classification module for receiving the set of action scores from the classification module and generating an action-agnostic start probability that the first video frame contains an action start. A fusion component is coupled to the localization module and the localization module for generating, based on the set of action scores and the action-agnostic start probability, a set of action-specific start probabilities, each action-specific start probability corresponding to a start of an action belonging to the respective action class.

公开(公告)号：US10699060B2

公开(公告)日：2020-06-30

申请号：US16000638

申请日：2018-06-05

申请人： salesforce.com, inc.

发明人： Bryan McCann ,  Caiming Xiong ,  Richard Socher

IPC分类号： G06F40/126 ,  G06N3/08 ,  G06N3/04 ,  G06F40/30 ,  G06F40/47 ,  G06F40/205 ,  G06F40/289 ,  G06F40/44 ,  G06F40/58

摘要： A system includes a neural network for performing a first natural language processing task. The neural network includes a first rectifier linear unit capable of executing an activation function on a first input related to a first word sequence, and a second rectifier linear unit capable of executing an activation function on a second input related to a second word sequence. A first encoder is capable of receiving the result from the first rectifier linear unit and generating a first task specific representation relating to the first word sequence, and a second encoder is capable of receiving the result from the second rectifier linear unit and generating a second task specific representation relating to the second word sequence. A biattention mechanism is capable of computing, based on the first and second task specific representations, an interdependent representation related to the first and second word sequences. In some embodiments, the first natural processing task performed by the neural network is one of sentiment classification and entailment classification.