CaltechAUTHORS
  A Caltech Library Service

Systems and methods for the determining annotator performance in the distributed annotation of source data

Welinder, Peter and Perona, Pietro (2018) Systems and methods for the determining annotator performance in the distributed annotation of source data. Patent number: US9898701B2. http://resolver.caltech.edu/CaltechAUTHORS:20180613-144511351

[img] PDF (US9898701B2) - Published Version
See Usage Policy.

51Kb

Use this Persistent URL to link to this item: http://resolver.caltech.edu/CaltechAUTHORS:20180613-144511351

Abstract

Systems and methods for determining annotator performance in the distributed annotation of source data in accordance embodiments of the invention are disclosed. In one embodiment of the invention, a method for clustering annotators includes obtaining a set of source data, determining a training data set representative of the set of source data, obtaining sets of annotations from a set of annotators for a portion of the training data set, for each annotator determining annotator recall metadata based on the set of annotations provided by the annotator for the training data set and determining annotator precision metadata based on the set of annotations provided by the annotator for the training data set, and grouping the annotators into annotator groups based on the annotator recall metadata and the annotator precision metadata.


Item Type:Patent
Related Patents:US20160275418A1
Patent Classification:Current U.S. Class: 1/1. Current CPC Class: G06F 17/30598 (20130101); G06F 17/30525 (20130101); G06F 17/30303 (20130101); G06N 5/048 (20130101); G06N 99/005 (20130101); G06F 17/241 (20130101); H05K 999/99 (20130101). Current International Class: G06F 17/30 (20060101); G06N 99/00 (20100101); G06N 5/04 (20060101); G06F 17/24 (20060101).
Patent Assignee:California Institute of Technology
Related URLs:
URLURL TypeDescription
https://patents.google.com/patent/US9898701B2/en?oq=US9898701B2PatentGoogle Patents
http://patft.uspto.gov/netacgi/nph-Parser?Sect2=PTO1&Sect2=HITOFF&p=1&u=/netahtml/PTO/search-bool.html&r=1&f=G&l=50&d=PALL&RefSrch=yes&Query=PN/9898701PatentU.S. Patent Office
ORCID:
AuthorORCID
Perona, Pietro0000-0002-7583-5809
Additional Information:Application filed: May 27, 2016. Application published: September 22, 2016. Patent granted: February 20, 2018. STATEMENT OF FEDERAL FUNDING: This invention was made with government support under Grant No. IIS0413312 awarded by the National Science Foundation and under Grant No. N00014-06-1-0734 and Grant No. N00014-10-1-0933 awarded by the Office of Naval Research. The government has certain rights in the invention. CROSS-REFERENCE TO RELATED APPLICATIONS: The current application is a continuation of U.S. patent application Ser. No. 13/921,493, filed Jun. 19, 2013 and issued as U.S. Pat. No. 9,355,360 on May 31, 2016, which claims priority to U.S. Provisional Patent Application No. 61/663,138, titled "Method for Combining Human and Machine Computation for Classification and Regression Tasks" to Welinder et al. and filed Jun. 22, 2012, the disclosures of which are hereby incorporated by reference in their entirety.
Funders:
Funding AgencyGrant Number
NSFIIS0413312
Office of Naval Research (ONR)N00014-06-1-0734
Office of Naval Research (ONR)N00014-10-1-0933
Record Number:CaltechAUTHORS:20180613-144511351
Persistent URL:http://resolver.caltech.edu/CaltechAUTHORS:20180613-144511351
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:87079
Collection:CaltechAUTHORS
Deposited By: Caroline Murphy
Deposited On:05 Jul 2018 23:36
Last Modified:05 Jul 2018 23:39

Repository Staff Only: item control page