公开(公告)号：US11238488B2

公开(公告)日：2022-02-01

申请号：US16824643

申请日：2020-03-19

Applicant: Microsoft Technology Licensing, LLC

Inventor： Min Liu ,  Parvez Ahammad

IPC: G06Q30/00 ,  G06Q30/02 ,  G06Q10/06

Abstract: A delayed grouping (batch) processing of previous campaign delivery pacing decisions and corresponding outcomes (deliveries) is used to configure a new auction experiment iteration. In the new iteration, a campaign that was previously over-delivered is classified as either (a) over-delivered due to incorrect pacing or (b) over-delivered due to auction experiment design. After the delayed processing, the new auction experiment iteration is conducted with a mitigating action taken on the previously over-delivered campaign if the campaign is classified as (b) over-delivered due to auction experiment design. For example, the mitigating action can include removing the campaign from a subsequent iteration of the experiment, or the experiment can be redesigned. By doing so, the over-delivery caused by the campaign due to the auction experiment design is avoided when performing the new auction experiment iteration.

公开(公告)号：US20210295374A1

公开(公告)日：2021-09-23

申请号：US16824643

申请日：2020-03-19

Applicant: Microsoft Technology Licensing, LLC

Inventor： Min Liu ,  Parvez Ahammad

IPC: G06Q30/02 ,  G06Q10/06

Abstract: A delayed grouping (batch) processing of previous campaign delivery pacing decisions and corresponding outcomes (deliveries) is used to configure a new auction experiment iteration. In the new iteration, a campaign that was previously over-delivered is classified as either (a) over-delivered due to incorrect pacing or (b) over-delivered due to auction experiment design. After the delayed processing, the new auction experiment iteration is conducted with a mitigating action taken on the previously over-delivered campaign if the campaign is classified as (b) over-delivered due to auction experiment design. For example, the mitigating action can include removing the campaign from a subsequent iteration of the experiment, or the experiment can be redesigned. By doing so, the over-delivery caused by the campaign due to the auction experiment design is avoided when performing the new auction experiment iteration.

公开(公告)号：US20210406598A1

公开(公告)日：2021-12-30

申请号：US16916706

申请日：2020-06-30

Applicant: Microsoft Technology Licensing, LLC

Inventor： Jilei Yang ,  Yu Liu ,  Parvez Ahammad ,  Fangfang Tan

IPC: G06K9/62 ,  G06N20/00

Abstract: Techniques for detecting label shift and adjusting training data of predictive models in response are provided. In an embodiment, a first machine-learned model is used to generate a predicted label for each of multiple scoring instances. The first machine-learned model is trained using one or more machine learning techniques based on a plurality of training instances, each of which includes an observed label. In response to detecting a shift in observed labels, for each segment of one or more segments in multiple segments, a portion of training data that corresponds to the segment is identified. For each training instance in a subset of the portion of training data, the training instance is adjusted. The adjusted training instance is added to a final set of training data. The machine learning technique(s) are used to train a second machine-learned model based on the final set of training data.

公开(公告)号：US20240143416A1

公开(公告)日：2024-05-02

申请号：US17978933

申请日：2022-11-01

Applicant: Microsoft Technology Licensing, LLC

Inventor： Ryan M. Rogers ,  Man Chun D. Leung ,  David Pardoe ,  Bing Liu ,  Shawn F. Ren ,  Rahul Tandra ,  Parvez Ahammad ,  Jing Wang ,  Ryan T. Tecco ,  Yajun Wang

IPC: G06F9/54 ,  G06F21/64

CPC classification number: G06F9/54 ,  G06F21/645

Abstract: Embodiments of the disclosed technologies receive first event data associated with a first party application, receive second event data representing a click, in the first party application, on a link to a third party application, receive third event data from the third party application, convert the third event data to a label, map a compressed format of the labeled third event data to the first event data and the second event data to create multi-party attribution data, group multiple instances of the multi-party attribution data into a batch, add noise to the compressed format of the labeled third event data in the batch, and send the noisy batch to a second computing device. A debiasing algorithm can be applied to the noisy batch. The debiased noisy batch can be used to train at least one machine learning model.

公开(公告)号：US20230319110A1

公开(公告)日：2023-10-05

申请号：US17709318

申请日：2022-03-30

Applicant: Microsoft Technology Licensing, LLC

Inventor： Ryan M. Rogers ,  Subbu Subramaniam ,  Mark B. Cesar ,  Adrian Rivera Cardoso ,  Yu Chen ,  Jefferson Lai ,  Vinyas Maddi ,  Lin Xu ,  Gavin Castro Uathavikul ,  Neha Jain ,  Shraddha Sahay ,  Parvez Ahammad ,  Rahul Tandra

IPC: H04L9/40

CPC classification number: H04L63/20

Abstract: Technologies for providing event-level data privacy for streaming post analytics data include, in some embodiments, receiving a data stream that includes instances of count data collected over a time interval, computing a true count breakdown that includes a set of sub-counts of non-public user interface interactions on the post, creating a noisy count breakdown by applying at least one differential privacy mechanism to the set of sub-counts, and streaming the noisy count breakdown instead of the true count breakdown to a computing device. At least one of the sub-counts is a count that is associated with a particular value of an attribute that has different possible values. The attribute is associated with the non-public user interface interactions on the post.

公开(公告)号：US11599746B2

公开(公告)日：2023-03-07

申请号：US16916706

申请日：2020-06-30

Applicant: Microsoft Technology Licensing, LLC

Inventor： Jilei Yang ,  Yu Liu ,  Parvez Ahammad ,  Fangfang Tan

IPC: G06N20/00 ,  G06V10/75 ,  G06K9/62

Abstract: Techniques for detecting label shift and adjusting training data of predictive models in response are provided. In an embodiment, a first machine-learned model is used to generate a predicted label for each of multiple scoring instances. The first machine-learned model is trained using one or more machine learning techniques based on a plurality of training instances, each of which includes an observed label. In response to detecting a shift in observed labels, for each segment of one or more segments in multiple segments, a portion of training data that corresponds to the segment is identified. For each training instance in a subset of the portion of training data, the training instance is adjusted. The adjusted training instance is added to a final set of training data. The machine learning technique(s) are used to train a second machine-learned model based on the final set of training data.

公开(公告)号：US11968236B2

公开(公告)日：2024-04-23

申请号：US17709318

申请日：2022-03-30

Applicant: Microsoft Technology Licensing, LLC

Inventor： Ryan M. Rogers ,  Subbu Subramaniam ,  Mark B. Cesar ,  Adrian Rivera Cardoso ,  Yu Chen ,  Jefferson Lai ,  Vinyas Maddi ,  Lin Xu ,  Gavin Castro Uathavikul ,  Neha Jain ,  Shraddha Sahay ,  Parvez Ahammad ,  Rahul Tandra

IPC: H04L9/40

CPC classification number: H04L63/20

Abstract: Technologies for providing event-level data privacy for streaming post analytics data include, in some embodiments, receiving a data stream that includes instances of count data collected over a time interval, computing a true count breakdown that includes a set of sub-counts of non-public user interface interactions on the post, creating a noisy count breakdown by applying at least one differential privacy mechanism to the set of sub-counts, and streaming the noisy count breakdown instead of the true count breakdown to a computing device. At least one of the sub-counts is a count that is associated with a particular value of an attribute that has different possible values. The attribute is associated with the non-public user interface interactions on the post.