A Caltech Library Service

Lean Multiclass Crowdsourcing

Van Horn, Grant and Branson, Steve and Loarie, Scott and Belongie, Serge and Perona, Pietro (2018) Lean Multiclass Crowdsourcing. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE , Piscataway, NJ, pp. 2714-2723. ISBN 978-1-5386-6420-9.

[img] PDF - Accepted Version
See Usage Policy.


Use this Persistent URL to link to this item:


We introduce a method for efficiently crowdsourcing multiclass annotations in challenging, real world image datasets. Our method is designed to minimize the number of human annotations that are necessary to achieve a desired level of confidence on class labels. It is based on combining models of worker behavior with computer vision. Our method is general: it can handle a large number of classes, worker labels that come from a taxonomy rather than a flat list, and can model the dependence of labels when workers can see a history of previous annotations. Our method may be used as a drop-in replacement for the majority vote algorithms used in online crowdsourcing services that aggregate multiple human annotations into a final consolidated label. In experiments conducted on two real-life applications we find that our method can reduce the number of required annotations by as much as a factor of 5.4 and can reduce the residual annotation error by up to 90% when compared with majority voting. Furthermore, the online risk estimates of the models may be used to sort the annotated collection and minimize subsequent expert review effort.

Item Type:Book Section
Related URLs:
URLURL TypeDescription
Belongie, Serge0000-0002-0388-5217
Perona, Pietro0000-0002-7583-5809
Additional Information:© 2018 IEEE. This work was supported by a Google Focused Research Award. We thank Oisin Mac Aodha for useful discussions.
Funding AgencyGrant Number
Record Number:CaltechAUTHORS:20180625-122336076
Persistent URL:
Official Citation:G. V. Horn, S. Branson, S. Loarie, S. Belongie and P. Perona, "Lean Multiclass Crowdsourcing," 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018, pp. 2714-2723. doi: 10.1109/CVPR.2018.00287
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:87336
Deposited By: Caroline Murphy
Deposited On:26 Jun 2018 19:50
Last Modified:15 Nov 2021 20:47

Repository Staff Only: item control page