CaltechAUTHORS
  A Caltech Library Service

Lean Crowdsourcing: Combining Humans and Machines in an Online System

Branson, Steve and Van Horn, Grant and Perona, Pietro (2017) Lean Crowdsourcing: Combining Humans and Machines in an Online System. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Conference on Computer Vision and Pattern Recognition . IEEE , New York, NY, pp. 6109-6118. ISBN 978-1-5386-0457-1. http://resolver.caltech.edu/CaltechAUTHORS:20180111-152555984

Full text is not posted in this repository. Consult Related URLs below.

Use this Persistent URL to link to this item: http://resolver.caltech.edu/CaltechAUTHORS:20180111-152555984

Abstract

We introduce a method to greatly reduce the amount of redundant annotations required when crowdsourcing annotations such as bounding boxes, parts, and class labels. For example, if two Mechanical Turkers happen to click on the same pixel location when annotating a part in a given image-an event that is very unlikely to occur by random chance-, it is a strong indication that the location is correct. A similar type of confidence can be obtained if a single Turker happened to agree with a computer vision estimate. We thus incrementally collect a variable number of worker annotations per image based on online estimates of confidence. This is done using a sequential estimation of risk over a probabilistic model that combines worker skill, image difficulty, and an incrementally trained computer vision model. We develop specialized models and algorithms for binary annotation, part keypoint annotation, and sets of bounding box annotations. We show that our method can reduce annotation time by a factor of 4-11 for binary filtering of websearch results, 2-4 for annotation of boxes of pedestrians in images, while in many cases also reducing annotation error. We will make an end-to-end version of our system publicly available.


Item Type:Book Section
Related URLs:
URLURL TypeDescription
http://dx.doi.org/10.1109/CVPR.2017.647DOIArticle
http://ieeexplore.ieee.org/document/8100130/PublisherArticle
ORCID:
AuthorORCID
Perona, Pietro0000-0002-7583-5809
Additional Information:© 2017 IEEE. Date Added to IEEE Xplore: 09 November 2017. This paper was inspired by work from and earlier collaborations with Peter Welinder and Boris Babenko. Much thanks to Pall Gunnarsson for helping to develop an early version of the method. Thank you to David Hall for supplying data for bounding box experiments. This work was supported by a Google Focused Research Award and Office of Naval Research MURI N000141010933.
Funders:
Funding AgencyGrant Number
Google Focused Research AwardUNSPECIFIED
Office of Naval Research (ONR)N000141010933
Record Number:CaltechAUTHORS:20180111-152555984
Persistent URL:http://resolver.caltech.edu/CaltechAUTHORS:20180111-152555984
Official Citation:S. Branson, G. V. Horn and P. Perona, "Lean Crowdsourcing: Combining Humans and Machines in an Online System," 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, 2017, pp. 6109-6118. doi: 10.1109/CVPR.2017.647
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:84278
Collection:CaltechAUTHORS
Deposited By: Ruth Sustaita
Deposited On:11 Jan 2018 23:39
Last Modified:11 Jan 2018 23:39

Repository Staff Only: item control page