A Caltech Library Service

The multidimensional wisdom of crowds

Welinder, Peter and Branson, Steve and Perona, Pietro and Belongie, Serge (2011) The multidimensional wisdom of crowds. In: Advances in Neural Information Processing Systems 23 (NIPS 2010). Advances in Neural Information Processing Systems. No.23. Neural Information Processing Systems , La Jolla, CA, pp. 2424-2432. ISBN 1617823805.

PDF - Published Version
See Usage Policy.


Use this Persistent URL to link to this item:


Distributing labeling tasks among hundreds or thousands of annotators is an increasingly important method for annotating large datasets. We present a method for estimating the underlying value (e.g. the class) of each image from (noisy) annotations provided by multiple annotators. Our method is based on a model of the image formation and annotation process. Each image has different characteristics that are represented in an abstract Euclidean space. Each annotator is modeled as a multidimensional entity with variables representing competence, expertise and bias. This allows the model to discover and represent groups of annotators that have different sets of skills and knowledge, as well as groups of images that differ qualitatively. We find that our model predicts ground truth labels on both synthetic and real data more accurately than state of the art methods. Experiments also show that our model, starting from a set of binary labels, may discover rich information, such as different “schools of thought” amongst the annotators, and can group together images belonging to separate categories.

Item Type:Book Section
Related URLs:
URLURL TypeDescription
Perona, Pietro0000-0002-7583-5809
Belongie, Serge0000-0002-0388-5217
Additional Information:©2010 Neural Information Processing Systems. P.P. and P.W. were supported by ONR MURI Grant #N00014-06-1-0734 and EVOLUT.ONR2. S.B. was sup- ported by NSF CAREER Grant #0448615, NSF Grant AGS-0941760, ONR MURI Grant #N00014-08-1-0638, and a Google Research Award.
Funding AgencyGrant Number
Office of Naval Research (ONR) Multidisciplinary University Research Initiative (MURI)N00014-06-1-0734
Office of Naval Research (ONR)EVOLUT.ONR2.
Office of Naval Research (ONR)N00014-08-1-0638
Google Research AwardUNSPECIFIED
Series Name:Advances in Neural Information Processing Systems
Issue or Number:23
Record Number:CaltechAUTHORS:20140916-124026882
Persistent URL:
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:49746
Deposited By: Caroline Murphy
Deposited On:16 Sep 2014 23:12
Last Modified:09 Mar 2020 13:18

Repository Staff Only: item control page