公开(公告)号：US20220172021A1

公开(公告)日：2022-06-02

申请号：US17455181

申请日：2021-11-16

Applicant: Oracle International Corporation

Inventor： Cong Duy Vu Hoang ,  Thanh Tien Vu ,  Poorya Zaremoodi ,  Ying Xu ,  Vladislav Blinov ,  Yu-Heng Hong ,  Yakupitiyage Don Thanuja Samodhye Dharmasiri ,  Vishal Vishnoi ,  Elias Luqman Jalaluddin ,  Manish Parekh ,  Thanh Long Duong ,  Mark Edward Johnson

IPC: G06N3/04 ,  G06N3/08

Abstract: Disclosed herein are techniques for addressing an overconfidence problem associated with machine learning models in chatbot systems. For each layer of a plurality of layers of a machine learning model, a distribution of confidence scores is generated for a plurality of predictions with respect to an input utterance. A prediction is determined for each layer of the machine learning model based on the distribution of confidence scores generated for the layer. Based on the predictions, an overall prediction of the machine learning model is determined. A subset of the plurality of layers are iteratively processed to identify a layer whose assigned prediction satisfies a criterion. A confidence score associated with the assigned prediction of the layer of the machine learning model is assigned as an overall confidence score to be associated with the overall prediction of the machine learning model.

公开(公告)号：US20220171946A1

公开(公告)日：2022-06-02

申请号：US17456687

申请日：2021-11-29

Applicant: Oracle International Corporation

Inventor： Ying Xu ,  Poorya Zaremoodi ,  Thanh Tien Vu ,  Cong Duy Vu Hoang ,  Vladislav Blinov ,  Yu-Heng Hong ,  Yakupitiyage Don Thanuja Samodhye Dharmasiri ,  Vishal Vishnoi ,  Elias Luqman Jalaluddin ,  Manish Parekh ,  Thanh Long Duong ,  Mark Edward Johnson

IPC: G06F40/35 ,  H04L51/02 ,  G06N3/08

Abstract: Techniques for using enhanced logit values for classifying utterances and messages input to chatbot systems in natural language processing. A method can include a chatbot system receiving an utterance generated by a user interacting with the chatbot system and inputting the utterance into a machine-learning model including a series of network layers. A final network layer of the series of network layers can include a logit function. The machine-learning model can map a first probability for a resolvable class to a first logit value using the logit function. The machine-learning model can map a second probability for a unresolvable class to an enhanced logit value. The method can also include the chatbot system classifying the utterance as the resolvable class or the unresolvable class based on the first logit value and the enhanced logit value.

公开(公告)号：US20220171930A1

公开(公告)日：2022-06-02

申请号：US17452742

申请日：2021-10-28

Applicant: Oracle International Corporation

Inventor： Elias Luqman Jalaluddin ,  Vishal Vishnoi ,  Thanh Long Duong ,  Mark Edward Johnson ,  Poorya Zaremoodi ,  Gautam Singaraju ,  Ying Xu ,  Vladislav Blinov

IPC: G06F40/279 ,  H04L12/58

Abstract: Techniques for keyword data augmentation for training chatbot systems in natural language processing. In one particular aspect, a method is provided that includes receiving a training set of utterances for training a machine-learning model to identify one or more intents for one or more utterances, augmenting the training set of utterances with out-of-domain (OOD) examples. The augmenting includes: identifying keywords within utterances of the training set of utterances, generating a set of OOD examples with the identified keywords, filtering out OOD examples from the set of OOD examples that have a context substantially similar to context of the utterances of the training set of utterances, and incorporating the set of OOD examples without the filtered OOD examples into the training set of utterances to generate an augmented training set of utterances. Thereafter, the machine-learning model is trained using the augmented training set of utterances.

公开(公告)号：US20220100961A1

公开(公告)日：2022-03-31

申请号：US17490792

申请日：2021-09-30

Applicant: Oracle International Corporation

Inventor： Vishal Vishnoi ,  Xin Xu ,  Elias Luqman Jalaluddin ,  Srinivasa Phani Kumar Gadde ,  Crystal C. Pan ,  Mark Edward Johnson ,  Thanh Long Duong ,  Balakota Srinivas Vinnakota ,  Manish Parekh

IPC: G06F40/295 ,  G06F40/35 ,  G06F40/211 ,  G06F40/56 ,  G06N5/04

Abstract: Techniques for automatically switching between chatbot skills in the same domain. In one particular aspect, a method is provided that includes receiving an utterance from a user within a chatbot session, where a current skill context is a first skill and a current group context is a first group, inputting the utterance into a candidate skills model for the first group, obtaining, using the candidate skills model, a ranking of skills within the first group, determining, based on the ranking of skills, a second skill is a highest ranked skill, changing the current skill context of the chatbot session to the second skill, inputting the utterance into a candidate flows model for the second skill, obtaining, using the candidate flows model, a ranking of intents within the second skill that match the utterance, and determining, based on the ranking of intents, an intent that is a highest ranked intent.

公开(公告)号：US20210082425A1

公开(公告)日：2021-03-18

申请号：US16983950

申请日：2020-08-03

Applicant: Oracle International Corporation

Inventor： Mark Edward Johnson ,  Michael Rye Kennewick

IPC: G10L15/22 ,  G06F16/22 ,  G06F16/2455 ,  G06N5/04 ,  G06N20/00 ,  G10L13/00 ,  G10L15/30 ,  G10L15/06 ,  G10L15/18

Abstract: Techniques are described for training and executing a machine learning model using data derived from a database. A dialog system uses data from the database to generate related training data for natural language understanding applications. The generated training data is then used to train a machine learning model. This enables the dialog system to leverage a large amount of available data to speed up the training process as compared to conventional labeling techniques. The dialog system uses the trained machine learning model to identify a named entity from a received spoken utterance and generate and output a speech response based upon the identified named entity.

公开(公告)号：US20210081799A1

公开(公告)日：2021-03-18

申请号：US16938098

申请日：2020-07-24

Applicant: Oracle International Corporation

Inventor： Mark Edward Johnson

IPC: G06N3/08 ,  G06F40/205 ,  G06F40/30 ,  G06F40/295 ,  G06N3/04

Abstract: A model for a natural language understanding task is generated based on labeled data generated by a labeling model. The model for the natural language understanding task is smaller than the labeling model (i.e., with lower computational and memory requirements than the combined model), but with substantially the same performance as the labeling model. In some cases, the labeling model may be generated based on a large pre-trained model.

公开(公告)号：US20210074274A1

公开(公告)日：2021-03-11

申请号：US16992306

申请日：2020-08-13

Applicant: Oracle International Corporation

Inventor： Thanh Long Duong ,  Mark Edward Johnson

IPC: G10L15/18 ,  G06F40/226 ,  G10L15/22 ,  G10L15/16

Abstract: Disclosed herein are techniques for using a generative adversarial network (GAN) to train a semantic parser of a dialog system. A method described herein involves accessing seed data that includes seed tuples. Each seed tuple includes a respective seed utterance and a respective seed logical form corresponding to the respective seed utterance. The method further includes training a semantic parser and a discriminator in a GAN. The semantic parser learns to map utterances to logical forms based on output from the discriminator, and the discriminator learns to recognize authentic logical forms based on output from the semantic parser. The semantic parser may then be integrated into a dialog system.

公开(公告)号：US20210074269A1

公开(公告)日：2021-03-11

申请号：US17002229

申请日：2020-08-25

Applicant: Oracle International Corporation

Inventor： Thanh Long Duong ,  Mark Edward Johnson

IPC: G10L15/16 ,  G10L15/06 ,  G10L15/22 ,  G06N3/08 ,  G06N20/00

Abstract: Techniques described herein use backpropagation to train one or more machine learning (ML) models of a dialog system. For instance, a method includes accessing seed data that includes training tuples, where each training tuple comprising a respective logical form. The method includes converting the logical form of a training tuple to a converted logical form, by applying to the logical form a text-to-speech (TTS) subsystem, an automatic speech recognition (ASR) subsystem, and a semantic parser of a dialog system. The method includes determining a training signal by using an objective function to compare the converted logical form to the logical form. The method further includes training the TTS subsystem, the ASR subsystem, and the semantic parser via backpropagation based on the training signal. As a result of the training by backpropagation, the machine learning models are tuned work effectively together within a pipeline of the dialog system.

公开(公告)号：US20210074262A1

公开(公告)日：2021-03-11

申请号：US16992291

申请日：2020-08-13

Applicant: Oracle International Corporation

Inventor： Thanh Long Duong ,  Mark Edward Johnson

IPC: G10L15/01 ,  G10L15/22 ,  G10L15/16

Abstract: Some techniques described herein determine a correction model for a dialog system, such that the correction model corrects output from an automatic speech recognition (ASR) subsystem in the dialog system. A method described herein includes accessing training data. A first tuple of the training data includes an utterance, where the utterance is a textual representation of speech. The method further includes using an ASR subsystem of a dialog system to convert the utterance to an output utterance. The method further includes storing the output utterance in corrective training data that is based on the training data. The method further includes training a correction model based on the corrective training data, such that the correction model is configured to correct output from the ASR subsystem during operation of the dialog system.

公开(公告)号：US12299402B2

公开(公告)日：2025-05-13

申请号：US18659606

申请日：2024-05-09

Applicant: Oracle International Corporation

Inventor： Thanh Long Duong ,  Mark Edward Johnson ,  Vishal Vishnoi ,  Crystal C. Pan ,  Vladislav Blinov ,  Cong Duy Vu Hoang ,  Elias Luqman Jalaluddin ,  Duy Vu ,  Balakota Srinivas Vinnakota

IPC: G06F40/30 ,  G06F40/205 ,  G06F40/289 ,  G06N20/00 ,  H04L51/02

Abstract: The present disclosure relates to techniques for identifying out-of-domain utterances. One particular technique includes receiving an utterance and a target domain of a chatbot, generating a sentence embedding for the utterance, obtaining an embedding representation for each cluster of in-domain utterances associated with the target domain, predicting, using a metric learning model, a first probability that the utterance belongs to the target domain based on a similarity or difference between the sentence embedding and each embedding representation for each cluster, predicting, using an outlier detection model, a second probability that the utterance belongs to the target domain based on a determined distance or density deviation between the sentence embedding and embedding representations for neighboring clusters, evaluating the first probability and the second probability to determine a final probability, and classifying the utterance as in-domain or out-of-domain for the chatbot based on the final probability.