A Caltech Library Service

Online crowdsourcing: rating annotators and obtaining cost-effective labels

Welinder, Peter and Perona, Pietro (2010) Online crowdsourcing: rating annotators and obtaining cost-effective labels. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR) Workshop on Advancing Computer Vision with Humans in the Loop (ACVHL). IEEE , Piscataway, NJ, pp. 25-32. ISBN 978-1-4244-7029-7.

PDF - Accepted Version
See Usage Policy.


Use this Persistent URL to link to this item:


Labeling large datasets has become faster, cheaper, and easier with the advent of crowdsourcing services like Amazon Mechanical Turk. How can one trust the labels obtained from such services? We propose a model of the labeling process which includes label uncertainty, as well a multi-dimensional measure of the annotators’ ability. From the model we derive an online algorithm that estimates the most likely value of the labels and the annotator abilities. It finds and prioritizes experts when requesting labels, and actively excludes unreliable annotators. Based on labels already obtained, it dynamically chooses which images will be labeled next, and how many labels to request in order to achieve a desired level of confidence. Our algorithm is general and can handle binary, multi-valued, and continuous annotations (e.g. bounding boxes). Experiments on a dataset containing more than 50,000 labels show that our algorithm reduces the number of labels required, and thus the total cost of labeling, by a large factor while keeping error rates low on a variety of datasets.

Item Type:Book Section
Related URLs:
URLURL TypeDescription
Perona, Pietro0000-0002-7583-5809
Additional Information:©2010 IEEE. We thank Catherine Wah, Florian Schroff, Steve Branson, and Serge Belongie for motivation, discussions and help with the data collection. We also thank Piotr Dollar, Merrielle Spain, Michael Maire, and Kristen Grauman for helpful discussions and feedback. This work was supported by ONR MURI Grant #N00014-06-1-0734 and ONR/Evolution Grant #N00173-09-C-4005.
Funding AgencyGrant Number
ONR MURI#N00014-06-1-0734
ONR Evolution#N00173-09-C-4005
Subject Keywords:Visipedia
Record Number:CaltechAUTHORS:20140730-102210223
Persistent URL:
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:47669
Deposited By: Caroline Murphy
Deposited On:30 Jul 2014 18:52
Last Modified:03 Oct 2019 06:55

Repository Staff Only: item control page