MODEL TESTING USING TEST SAMPLE UNCERTAINTY
    3.
    发明公开

    公开(公告)号:US20240185027A1

    公开(公告)日:2024-06-06

    申请号:US18074148

    申请日:2022-12-02

    IPC分类号: G06N3/045 G06N3/08

    CPC分类号: G06N3/045 G06N3/08

    摘要: Using encoded representations of target model training data and a label corresponding to each portion of the target model training data, a proxy model to determine an uncertainty score corresponding to an output of a trained target model is trained. Using the trained proxy model, a set of uncertainty scores is computed, each uncertainty score in the set of uncertainty scores corresponding to a portion of target model testing data in a set of target model testing data. A subset of the set of target model testing data is selected, the subset comprising a plurality of portions of target model testing data having an uncertainty score above a threshold uncertainty score. Using the subset of the set of target model testing data, the trained target model.

    Automated testing of dialog systems

    公开(公告)号:US11645470B2

    公开(公告)日:2023-05-09

    申请号:US17136469

    申请日:2020-12-29

    摘要: Methods, systems and computer program products for automated testing of dialog systems are provided herein. A computer-implemented method includes receiving information pertaining to a given conversation workspace of an automated dialog system and identifying test case inputs to the automated dialog system, the test case inputs comprising user input for the given conversation workspace that has portions thereof modified and which the automated dialog system maps to a different intent and/or a different entity relative to the user input. The method further includes generating human-interpretable explanations of mappings of portions of the test case inputs to the different intent and/or entity, generating suggestions for modifying intents, entities and dialog flows of the given conversation workspace such that the test case inputs map to the same intent and/or the same entity as their corresponding user input, and outputting the suggestions and the human-interpretable explanations to a user.

    Natural language interface databases

    公开(公告)号:US11200222B2

    公开(公告)日:2021-12-14

    申请号:US16393090

    申请日:2019-04-24

    摘要: Embodiments are disclosed for correcting a natural language interface database (NLIDB) system. The techniques include receiving feedback indicating that an answer provided in response to a question for an NLIDB system is inaccurate. The techniques further include finding an ontology element for a datastore of the NLIDB system that matches to the feedback. The techniques also include selecting candidate annotations for the NLIDB system based on the ontology element and a data type of the ontology element. Additionally, the techniques include generating a question-answer (QA) pair for each of the candidate annotations. Further, the techniques include adding one of the candidate annotations to annotations for a natural language query (NLQ) engine of the NLIDB system based on a client verification of the QA pair.

    MULTI-AGENT CONVERSATIONAL AGENT FRAMEWORK

    公开(公告)号:US20210064706A1

    公开(公告)日:2021-03-04

    申请号:US16552557

    申请日:2019-08-27

    IPC分类号: G06F17/28 H04L12/58 G06Q30/08

    摘要: One embodiment provides a method, including: receiving, within a conversational agent creation framework used to create conversational agents that perform negotiations on behalf of users, (i) a selection of one of a plurality of conversational agent roles (ii) negotiation constraints for the selected role, and (iii) an object for the negotiation; creating a conversational agent (i) having the selected role and (ii) programmed with the negotiation constraints; identifying at least one other conversational agent (i) having an opposing role with respect to the object and (ii) being programmed with opposing role negotiation constraints; and conducting a conversational session between the conversational agent and the at least one other conversational agent, wherein the conversational session comprises the negotiation for the object and wherein the conversational agent and the at least one other conversational agent perform the negotiation in view of the negotiation constraints and the opposing role negotiation constraints.

    BIAS REDUCTION IN CROWDSOURCED TASKS

    公开(公告)号:US20210019349A1

    公开(公告)日:2021-01-21

    申请号:US16517044

    申请日:2019-07-19

    摘要: One embodiment provides a method, including: identifying at least one protected attribute of a task of annotating information, wherein the task is assigned to at least one crowdsourced worker for performance of the task; generating at least one question for detecting bias of the at least one crowdsourced worker with respect to the at least one protected attribute, the at least one question comprising a previously provided annotation; providing the at least one question to the at least one crowdsourced worker at a period during performance of the task; computing a bias of the at least one crowdsourced worker with respect to the at least one protected attribute by comparing (i) an annotation provided by the crowdsourced worker to the at least one question and (ii) the previously provided annotation; and updating a profile of the at least one crowdsourced worker with the computed bias.

    Ontology-Based Automatic Bootstrapping of State-Based Dialog Systems

    公开(公告)号:US20190228068A1

    公开(公告)日:2019-07-25

    申请号:US15875303

    申请日:2018-01-19

    IPC分类号: G06F17/27 G06F17/30

    摘要: Methods, systems, and computer program products for bootstrapping of state-based dialog systems are provided herein. A computer-implemented method includes determining parameters for state automata by partitioning an ontology graph into sub-graphs and a knowledge graph into sub-graphs, wherein the ontology graph and the knowledge graph are based on a user question and domain knowledge pertaining to the user question; generating a structured query for each of the sub-graphs; determining intentions of a dialog pertaining to the at least one user question by translating each of the generated structured queries to a respective natural language query; creating one or more dialog states for each of the determined dialog intentions; creating one or more connecting dialog states between pairs of the created dialog states; and generating an automata dialog framework associated with the user question based on the created dialog states and the created connecting dialog states.

    Interactive Dialog in Natural Language Using an Ontology

    公开(公告)号:US20180276273A1

    公开(公告)日:2018-09-27

    申请号:US15467291

    申请日:2017-03-23

    IPC分类号: G06F17/30 G06F17/27

    CPC分类号: G06F16/243 G06F17/2785

    摘要: Methods, systems, and computer program products for carrying out interactive dialog in natural language using an ontology are provided herein. A computer-implemented method includes generating one or more domain-driven interpretations of a natural language dialogue query provided by a user via utilization of a web ontology language; determining multiple structured base queries, from among a stored collection of structured queries, that correspond to the natural language dialogue query, in view of the one or more generated domain-driven interpretations; selecting one of the multiple determined structured base queries, based on one or more items of context information pertaining to the dialogue; automatically generating a response to the selected structured base query; and outputting the generated response to the user.