SYSTEM AND METHOD FOR EVALUATING GENERATIVE LARGE LANGUAGE MODELS

    公开(公告)号:US20250111237A1

    公开(公告)日:2025-04-03

    申请号:US18476632

    申请日:2023-09-28

    Abstract: Aspects of the subject disclosure may include, for example, a device that facilitates obtaining a plurality of prompts from a selected subject matter domain of a database configured to measure an effectiveness of a generative large language model (LLM) to distinguish variances between each prompt of the plurality of prompts; supplying the plurality of prompts to the LLM; receiving respective responses to each of the prompts from the LLM; transforming each of the prompts and respective responses to each of the prompts into an embedding space; determining, by applying domain-based metrics to the embedding space, a quality measurement of each respective response to produce a plurality of quality measurements; and generating, according to the plurality of quality measurements, a performance of the LLM. Other embodiments are disclosed.

    TESTING ACCESSIBILITY OF GEOSPATIAL APPLICATIONS USING CONTEXTUAL FILTERS

    公开(公告)号:US20250094161A1

    公开(公告)日:2025-03-20

    申请号:US18959330

    申请日:2024-11-25

    Abstract: A method performed by a processing system including at least one processor includes applying a contextual filter to mask a portion of at least one of: an input of a software application, an output of the software application, or an underlying dataset of the software application, where the contextual filter simulates a limitation of a user of the software application, executing the software application with the contextual filter applied to the at least one of: the input of the software application, the output of the software application, or the underlying dataset of the software application, collecting ambient data during the executing, and recommending, based on a result of the executing, a modification to the software application to improve at least one of: an accessibility of the software application or an inclusion of the software application.

    SIMULATING TRAINING DATA TO MITIGATE BIASES IN MACHINE LEARNING MODELS

    公开(公告)号:US20230350977A1

    公开(公告)日:2023-11-02

    申请号:US17661026

    申请日:2022-04-27

    CPC classification number: G06K9/6257 G06K9/6289 G06N20/00

    Abstract: A method performed by a processing system including at least one processor includes identifying an insufficiency in a representation of a subpopulation in training data for a machine learning model, generating simulated data to mitigate the insufficiency in the representation, and training the machine learning model using an enhanced training data set that includes the training data and the simulated data to produce a trained machine learning model. In some examples, the generating and the training may be repeated in response to determining that an output of the trained machine learning model still reflects the insufficiency in the representation of the subpopulation or reflects an insufficiency in a representation of another subpopulation. In other examples, the simulated data may be stored for future reuse.

    AUTOMATING BIAS EVALUATION FOR MACHINE LEARNING PROJECTS

    公开(公告)号:US20230267362A1

    公开(公告)日:2023-08-24

    申请号:US17652268

    申请日:2022-02-23

    CPC classification number: G06N20/00

    Abstract: A method includes obtaining descriptive information for a first machine learning project, identifying, based on the descriptive information, a plurality of past machine learning projects which are similar to the first machine learning project, retrieving digital documents that describe the bias evaluation pipelines that were used to evaluate the plurality of past machine learning projects, detecting a common bias evaluation pipeline step among at least a subset of the digital documents, extracting, from the subset, a snippet of machine-executable code that corresponds to the common bias evaluation pipeline step, modifying the snippet of machine-executable code with use case data that is specific to the first machine learning project to generate modified machine-executable code, and generating a proposed bias evaluation pipeline for evaluating the first machine learning project, wherein the proposed bias evaluation pipeline includes the modified machine-executable code.

    TRAINING DATA FIDELITY FOR MACHINE LEARNING APPLICATIONS THROUGH INTELLIGENT MERGER OF CURATED AUXILIARY DATA

    公开(公告)号:US20230057792A1

    公开(公告)日:2023-02-23

    申请号:US17408384

    申请日:2021-08-21

    Abstract: In one example, a method includes identifying a target performance metric of a machine learning algorithm, wherein the target performance metric is to be improved, obtaining a set of auxiliary data from a plurality of auxiliary data sources, wherein the plurality of auxiliary data sources is separate from a training data set used to train the machine learning algorithm, selecting a candidate attribute type from the set of auxiliary data, identifying a quality metric for the candidate attribute type, calculating a change in the target performance metric when data values associated with the candidate attribute type are included in the training data set, determining that a tradeoff between the target performance metric and the quality metric of the candidate attribute type is satisfied by inclusion of the data values in the training data set, and training the machine learning algorithm using the training data set augmented with the data value.

    METHOD AND APPARATUS TO IDENTIFY OUTLIERS IN SOCIAL NETWORKS

    公开(公告)号:US20170242926A1

    公开(公告)日:2017-08-24

    申请号:US15589072

    申请日:2017-05-08

    Abstract: A system that incorporates teachings of the present disclosure may include, for example, a process that reduces a sampling size of a total population of on-line social network users based on a comparison of seed information to a population of on-line social network users. The reduced sampling of on-line social network users is compared to a social graph of the on-line social network users, wherein the social graph is obtained from an algorithm applied to the reduced sampling of the on-line social network users. An outlier is determined in the reduced sampling of on-line social network users based on a characterizing of a cluster of social network users. Additional embodiments are disclosed.

    Bias scoring of machine learning project data

    公开(公告)号:US11983646B2

    公开(公告)日:2024-05-14

    申请号:US18172654

    申请日:2023-02-22

    CPC classification number: G06N5/04 G06F16/285 G06N20/00

    Abstract: Aspects of the subject disclosure may include, for example, system and apparatus that enable operations that may include receiving, by a processing system, project data defining a proposed machine learning (ML) project of an entity and storing the project data in a project database with other project data for other projects. The operations may further include extracting extracted features of the proposed project and, based on the extracted features, determining a clustering assignment for the proposed project. Determining the clustering assignment may comprise comparing information about the proposed project including the extracted features with information about the other projects and assigning the proposed project to a cluster including one or more projects having similar bias characteristics as the proposed project. The operations may further include determining a risk of potential bias for the proposed project and, based on the risk of bias, recommending a corrective action to reduce the risk of bias. Machine learning models may be used for project clustering and bias score determination and may be readily updated as new ML projects are evaluated. Other embodiments are disclosed.

    COMPENSATING FOR VULNERABILITIES IN MACHINE LEARNING ALGORITHMS

    公开(公告)号:US20230057593A1

    公开(公告)日:2023-02-23

    申请号:US17408385

    申请日:2021-08-21

    Abstract: A method performed by a processing system including at least one processor includes obtaining an output of a machine learning algorithm, identifying a vulnerability in the output of the machine learning algorithm, wherein the vulnerability relates to a bias in the output, integrating auxiliary data from an auxiliary data source of a plurality of auxiliary data sources into the machine learning algorithm to try to compensate for the vulnerability, determining whether the integrating has compensated for the vulnerability, and generating a runtime output using the machine learning algorithm when the processing system determines that the integrating has compensated for the vulnerability.

Patent Agency Ranking