公开(公告)号：US20190012302A1

公开(公告)日：2019-01-10

申请号：US14640591

申请日：2015-03-06

Applicant: Google Inc.

Inventor： Matt MacMahon ,  Tania Bedrax Weiss

IPC: G06F17/24 ,  G06F17/30

Abstract: Methods and apparatus related to verifying annotations of textual segments based on human feedback that is responsive to questions generated to solicit feedback relevant to the annotations. Some implementations are directed generally to generating one or more task specifications to solicit feedback relevant to a potential annotation of a target textual segment, transmitting the task specifications for review by a plurality of human reviewers, receiving feedback responsive to the task specifications, and using the feedback to determine whether the potential annotation is a verified annotation. Some implementations are directed generally to determining an effectiveness measure for a task system and/or one or more reviewers, wherein the effectiveness measure is indicative of effectiveness in providing feedback instances related to one or more annotations for one or more textual segments.