A Caltech Library Service

Systems and methods for distributed data annotation

Welinder, Peter and Perona, Pietro (2018) Systems and methods for distributed data annotation. Patent number: US9928278B2.

[img] PDF (US9928278B2) - Published Version
See Usage Policy.


Use this Persistent URL to link to this item:


Systems and methods for distributed data annotation in accordance embodiments of the invention are disclosed. In one embodiment of the invention, a distributed data annotation server system includes a storage device configured to store source data, one or more annotators, annotation tasks and a processor, wherein a distributed data annotation application configures the processor to receive source data including one or more pieces of source data, select one or more annotators, create one or more annotation tasks for the selected annotators and source data, request one or more annotations for the source data using the annotation tasks, receive annotations, determine source data metadata for at least one piece of source data using the received annotations, generate annotator metadata for at least one annotator using the received annotations and the source data, and estimate the ground truth for the source data using the source data metadata and the annotator metadata.

Item Type:Patent
Related Patents:US20140188879A1
Patent Classification:Current U.S. Class: 1/1. Current CPC Class: G06F 17/30997 (20130101); G06Q 10/101 (20130101); G06Q 50/24 (20130101); G06F 17/301 (20130101); G06F 17/30525 (20130101); G16H 10/60 (20180101); G06F 19/00 (20130101); H05K 999/99 (20130101); G06F 17/30864 (20130101); G06F 17/30011 (20130101); G06F 17/30867 (20130101). Current International Class: G06F 17/30 (20060101); G06Q 10/10 (20120101).
Patent Assignee:California Institute of Technology
Related URLs:
URLURL TypeDescription Patents Patent Office
Perona, Pietro0000-0002-7583-5809
Additional Information:Application filed: March 6, 2014. Application published: July 3, 2014. Patent granted: March 27, 2018. FEDERAL FUNDING SUPPORT: This invention was made with government support under AGS0941760 awarded by the National Science Foundation and under N00014-08-1-0638; N00014-06-1-0734 awarded by the NAVY. The government has certain rights in the invention. CROSS-REFERENCE TO RELATED APPLICATIONS: The current application is a continuation of U.S. patent application Ser. No. 13/651,108, filed Oct. 12, 2012, that claims priority to U.S. Provisional Patent Application No. 61/546,386, titled "Algorithm for Estimating for Annotator Expertise, Bias and Competence and Image Difficulty in Binary Image Annotation" and filed Oct. 12, 2011, the disclosures of which are incorporated herein by reference in their entirety.
Funding AgencyGrant Number
Office of Naval Research (ONR)N00014-08-1-0638
Office of Naval Research (ONR)N00014-06-1-0734
Record Number:CaltechAUTHORS:20180613-142547321
Persistent URL:
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:87077
Deposited By: Caroline Murphy
Deposited On:05 Jul 2018 23:45
Last Modified:03 Oct 2019 19:52

Repository Staff Only: item control page