Welinder, Peter and Branson, Steve and Perona, Pietro and Belongie, Serge (2011) The multidimensional wisdom of crowds. In: Advances in Neural Information Processing Systems 23 (NIPS 2010). Advances in Neural Information Processing Systems. No.23. Neural Information Processing Systems , La Jolla, CA, pp. 2424-2432. ISBN 1617823805. https://resolver.caltech.edu/CaltechAUTHORS:20140916-124026882
|
PDF
- Published Version
See Usage Policy. 2467Kb |
Use this Persistent URL to link to this item: https://resolver.caltech.edu/CaltechAUTHORS:20140916-124026882
Abstract
Distributing labeling tasks among hundreds or thousands of annotators is an increasingly important method for annotating large datasets. We present a method for estimating the underlying value (e.g. the class) of each image from (noisy) annotations provided by multiple annotators. Our method is based on a model of the image formation and annotation process. Each image has different characteristics that are represented in an abstract Euclidean space. Each annotator is modeled as a multidimensional entity with variables representing competence, expertise and bias. This allows the model to discover and represent groups of annotators that have different sets of skills and knowledge, as well as groups of images that differ qualitatively. We find that our model predicts ground truth labels on both synthetic and real data more accurately than state of the art methods. Experiments also show that our model, starting from a set of binary labels, may discover rich information, such as different “schools of thought” amongst the annotators, and can group together images belonging to separate categories.
Item Type: | Book Section | ||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Related URLs: |
| ||||||||||||||
ORCID: |
| ||||||||||||||
Additional Information: | ©2010 Neural Information Processing Systems. P.P. and P.W. were supported by ONR MURI Grant #N00014-06-1-0734 and EVOLUT.ONR2. S.B. was sup- ported by NSF CAREER Grant #0448615, NSF Grant AGS-0941760, ONR MURI Grant #N00014-08-1-0638, and a Google Research Award. | ||||||||||||||
Funders: |
| ||||||||||||||
Series Name: | Advances in Neural Information Processing Systems | ||||||||||||||
Issue or Number: | 23 | ||||||||||||||
Record Number: | CaltechAUTHORS:20140916-124026882 | ||||||||||||||
Persistent URL: | https://resolver.caltech.edu/CaltechAUTHORS:20140916-124026882 | ||||||||||||||
Usage Policy: | No commercial reproduction, distribution, display or performance rights in this work are provided. | ||||||||||||||
ID Code: | 49746 | ||||||||||||||
Collection: | CaltechAUTHORS | ||||||||||||||
Deposited By: | Caroline Murphy | ||||||||||||||
Deposited On: | 16 Sep 2014 23:12 | ||||||||||||||
Last Modified: | 09 Mar 2020 13:18 |
Repository Staff Only: item control page