CaltechAUTHORS
  A Caltech Library Service

Learning From Noisy Singly-labeled Data

Khetan, Ashish and Lipton, Zachary C. and Anandkumar, Animashree (2018) Learning From Noisy Singly-labeled Data. In: 6th International Conference on Learning Representations (ICLR 2018), 30 April-3 May 2018, Vancouver, Canada. https://resolver.caltech.edu/CaltechAUTHORS:20190327-085735866

[img] PDF - Published Version
See Usage Policy.

407Kb

Use this Persistent URL to link to this item: https://resolver.caltech.edu/CaltechAUTHORS:20190327-085735866

Abstract

Supervised learning depends on annotated examples, which are taken to be the ground truth. But these labels often come from noisy crowdsourcing platforms, like Amazon Mechanical Turk. Practitioners typically collect multiple labels per example and aggregate the results to mitigate noise (the classic crowdsourcing problem). Given a fixed annotation budget and unlimited unlabeled data, redundant annotation comes at the expense of fewer labeled examples. This raises two fundamental questions: (1) How can we best learn from noisy workers? (2) How should we allocate our labeling budget to maximize the performance of a classifier? We propose a new algorithm for jointly modeling labels and worker quality from noisy crowd-sourced data. The alternating minimization proceeds in rounds, estimating worker quality from disagreement with the current model and then updating the model by optimizing a loss function that accounts for the current estimate of worker quality. Unlike previous approaches, even with only one annotation per example, our algorithm can estimate worker quality. We establish a generalization error bound for models learned with our algorithm and establish theoretically that it's better to label many examples once (vs less multiply) when worker quality exceeds a threshold. Experiments conducted on both ImageNet (with simulated noisy workers) and MS-COCO (using the real crowdsourced labels) confirm our algorithm's benefits.


Item Type:Conference or Workshop Item (Poster)
Related URLs:
URLURL TypeDescription
https://openreview.net/forum?id=H1sUHgb0ZPublisherArticle
http://arxiv.org/abs/1712.04577arXivArticle
Record Number:CaltechAUTHORS:20190327-085735866
Persistent URL:https://resolver.caltech.edu/CaltechAUTHORS:20190327-085735866
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:94170
Collection:CaltechAUTHORS
Deposited By: George Porter
Deposited On:29 Mar 2019 20:17
Last Modified:03 Oct 2019 21:01

Repository Staff Only: item control page