公开(公告)号：US20230109681A1

公开(公告)日：2023-04-13

申请号：US17587984

申请日：2022-01-28

Applicant: salesforce.com, inc.

Inventor： Akhilesh Deepak Gotmare ,  Junnan Li ,  Chu Hong Hoi

IPC: G06F40/151 ,  G06F40/30 ,  G06F40/40 ,  G06N3/04

Abstract: Embodiments are directed to translating a natural language query into a code snippet in a programing language that semantically represents the query. The embodiments include a cascading neural network that includes an encoder network and a classifier network. The encoder network being faster but less accurate than the classifier network. The encoder network is trained using a contrastive learning framework to identify code candidates from a large set of code snippets. The classifier network is trained using a binary classifier to identify the code snippet that semantically represents the query from the code candidates.

公开(公告)号：US20220391755A1

公开(公告)日：2022-12-08

申请号：US17370524

申请日：2021-07-08

Applicant: salesforce.com, inc.

Inventor： Junnan Li ,  Chu Hong Hoi

IPC: G06N20/00 ,  G06K9/62 ,  G06F40/20 ,  G06K9/34 ,  G06N3/02

Abstract: Embodiments described herein provide visual-and-language (V+L) systems and methods for learning vision and language representations. Specifically, a method may comprise receiving a training dataset comprising a plurality of image samples and a plurality of text samples; encoding the plurality of image samples into a plurality of encoded image samples and the plurality of text samples into a plurality of encoded text samples; computing a first loss objective based on the plurality of encoded image samples and the plurality of encoded text samples; encoding a first subset of the plurality of encoded image samples and a second subset of the plurality of encoded text samples into a plurality of encoded image-text samples; computing a second loss objective based on the plurality of encoded image-text samples; and updating the V+L model based at least in part on the first loss objective and the second loss objective.

公开(公告)号：US20220156593A1

公开(公告)日：2022-05-19

申请号：US17219339

申请日：2021-03-31

Applicant: salesforce.com, inc.

Inventor： Hualin Liu ,  Chu Hong Hoi ,  Junnan Li

IPC: G06N3/08 ,  G06K9/62

Abstract: Embodiments described herein provide systems and methods for learning representation from unlabeled videos. Specifically, a method may comprise generating a set of strongly-augmented samples and a set of weakly-augmented samples from the unlabeled video samples; generating a set of predictive logits by inputting the set of strongly-augmented samples into a student model and a first teacher model; generating a set of artificial labels by inputting the set of weakly-augmented samples to a second teacher model that operates in parallel to the first teacher model, wherein the second teacher model shares one or more model parameters with the first teacher model; computing a loss objective based on the set of predictive logits and the set of artificial labels; updating student model parameters based on the loss objective via backpropagation; and updating the shared parameters for the first teacher model and the second teacher model based on the updated student model parameters.

公开(公告)号：US11599792B2

公开(公告)日：2023-03-07

申请号：US16688104

申请日：2019-11-19

Applicant: salesforce.com, inc.

Inventor： Junnan Li ,  Chu Hong Hoi

IPC: G06N3/08 ,  G06N3/04

Abstract: A method provides learning with noisy labels. The method includes generating a first network of a machine learning model with a first set of parameter initial values, and generating a second network of the machine learning model with a second set of parameter initial values. First clean probabilities for samples in a training dataset are generated using the second network. A first labeled dataset and a first unlabeled dataset are generated from the training dataset based on the first clean probabilities. The first network is trained based on the first labeled dataset and first unlabeled dataset to update parameters of the first network.

公开(公告)号：US11334766B2

公开(公告)日：2022-05-17

申请号：US16778339

申请日：2020-01-31

Applicant: salesforce.com, inc.

Inventor： Junnan Li ,  Chu Hong Hoi

IPC: G06K9/62 ,  G06N3/04 ,  G06N3/08

Abstract: Systems and methods are provided for training object detectors of a neural network model with a mixture of label noise and bounding box noise. According to some embodiments, a learning framework is provided which jointly optimizes object labels, bounding box coordinates, and model parameters by performing alternating noise correction and model training. In some embodiments, to disentangle label noise and bounding box noise, a two-step noise correction method is employed. In some examples, the first step performs class-agnostic bounding box correction by minimizing classifier discrepancy and maximizing region objectness. In some examples, the second step uses dual detection heads for label correction and class-specific bounding box refinement.

公开(公告)号：US20220067506A1

公开(公告)日：2022-03-03

申请号：US17005763

申请日：2020-08-28

Applicant: salesforce.com, inc.

Inventor： Junnan Li ,  Chu Hong Hoi

IPC: G06N3/08 ,  G06K9/62

Abstract: A learning mechanism with partially-labeled web images is provided while correcting the noise labels during the learning. Specifically, the mechanism employs a momentum prototype that represents common characteristics of a specific class. One training objective is to minimize the difference between the normalized embedding of a training image sample and the momentum prototype of the corresponding class. Meanwhile, during the training process, the momentum prototype is used to generate a pseudo label for the training image sample, which can then be used to identify and remove out of distribution (OOD) samples to correct the noisy labels from the original partially-labeled training images. The momentum prototype for each class is in turn constantly updated based on the embeddings of new training samples and their pseudo labels.

公开(公告)号：US20230162490A1

公开(公告)日：2023-05-25

申请号：US17589725

申请日：2022-01-31

Applicant: salesforce.com, inc.

Inventor： Shu Zhang ,  Junnan Li ,  Ran Xu ,  Caiming Xiong ,  Chetan Ramaiah

IPC: G06V10/776 ,  G06V10/74 ,  G06F40/284 ,  G06F40/166 ,  G06F40/126 ,  G06V10/80 ,  G06F16/583 ,  G06F16/56

CPC classification number: G06V10/776 ,  G06V10/761 ,  G06F40/284 ,  G06F40/166 ,  G06F40/126 ,  G06V10/806 ,  G06F16/5846 ,  G06F16/56

Abstract: Embodiments described herein a CROss-Modal Distribution Alignment (CROMDA) model for vision-language pretraining, which can be used for retrieval downstream tasks. In the CROMDA mode, global cross-modal representations are aligned on each unimodality. Specifically, a uni-modal global similarity between an image/text and the image/text feature queue are computed. A softmax-normalized distribution is then generated based on the computed similarity. The distribution thus takes advantage of property of the global structure of the queue. CROMDA then aligns the two distributions and learns a modal invariant global representation. In this way, CROMDA is able to obtain invariant property in each modality, where images with similar text representations should be similar and vice versa.

公开(公告)号：US20230154146A1

公开(公告)日：2023-05-18

申请号：US17566061

申请日：2021-12-30

Applicant: salesforce.com, inc.

Inventor： Dongxu Li ,  Junnan Li ,  Chu Hong Hoi

IPC: G06V10/74 ,  G06V10/774 ,  G06F40/279 ,  G06V20/40 ,  G06V10/776

CPC classification number: G06V10/761 ,  G06V10/774 ,  G06F40/279 ,  G06V20/47 ,  G06V20/41 ,  G06V10/776 ,  G06V20/46

Abstract: Embodiments described a method of video-text pre-learning to effectively learn cross-modal representations from sparse video frames and text. Specifically, an align and prompt framework provides a video and language pre-training framework that encodes the frames and text independently using a transformer-based video encoder and a text encoder. A multi-modal encoder is then employed to capture cross-modal interaction between a plurality of video frames and a plurality of texts. The pre-training includes a prompting entity modeling that enables the model to capture fine-grained region-entity alignment.

公开(公告)号：US11263476B2

公开(公告)日：2022-03-01

申请号：US16870621

申请日：2020-05-08

Applicant: salesforce.com, inc.

Inventor： Junnan Li ,  Chu Hong Hoi

IPC: G06K9/62 ,  G06T7/73

Abstract: The system and method are directed to a prototypical contrastive learning (PCL). The PCL explicitly encodes the hierarchical semantic structure of the dataset into the learned embedding space and prevents the network from exploiting low-level cues for solving the unsupervised learning task. The PCL includes prototypes as the latent variables to help find the maximum-likelihood estimation of the network parameters in an expectation-maximization framework. The PCL iteratively performs an E-step for finding prototypes with clustering and M-step for optimizing the network on a contrastive loss.

公开(公告)号：US20210374553A1

公开(公告)日：2021-12-02

申请号：US17015858

申请日：2020-09-09

Applicant: salesforce.com, inc.

Inventor： Junnan Li ,  Chu Hong Hoi

IPC: G06N3/08 ,  G06N3/04

Abstract: Embodiments described herein provide systems and methods for noise-robust contrastive learning. In view of the need for a noise-robust learning system, embodiments described herein provides a contrastive learning mechanism that combats noise by learning robust representations of the noisy data samples. Specifically, the training images are projected into a low-dimensional subspace, and the geometric structure of the subspace is regularized with: (1) a consistency contrastive loss that enforces images with perturbations to have similar embeddings; and (2) a prototypical contrastive loss augmented with a predetermined learning principle, which encourages the embedding for a linearly-interpolated input to have the same linear relationship with respect to the class prototypes. The low-dimensional embeddings are also trained to reconstruct the high-dimensional features, which preserves the learned information and regularizes the classifier.