CaltechAUTHORS
  A Caltech Library Service

Active Learning under Label Shift

Zhao, Eric and Liu, Anqi and Anandkumar, Animashree and Yue, Yisong (2020) Active Learning under Label Shift. . (Unpublished) https://resolver.caltech.edu/CaltechAUTHORS:20201110-074357009

[img] PDF - Submitted Version
See Usage Policy.

4Mb

Use this Persistent URL to link to this item: https://resolver.caltech.edu/CaltechAUTHORS:20201110-074357009

Abstract

Distribution shift poses a challenge for active data collection in the real world. We address the problem of active learning under label shift and propose ALLS, the first framework for active learning under label shift. ALLS builds on label shift estimation techniques to correct for label shift with a balance of importance weighting and class-balanced sampling. We show a bias-variance trade-off between these two techniques and prove error and sample complexity bounds for a disagreement-based algorithm under ALLS. Experiments across a range of label shift settings demonstrate ALLS consistently improves performance, often reducing sample complexity by more than half an order of magnitude. Ablation studies corroborate the bias-variance trade-off revealed by our theory.


Item Type:Report or Paper (Discussion Paper)
Related URLs:
URLURL TypeDescription
http://arxiv.org/abs/2007.08479arXivDiscussion Paper
ORCID:
AuthorORCID
Zhao, Eric0000-0002-9595-0150
Yue, Yisong0000-0001-9127-1989
Additional Information:Anqi Liu is supported by PIMCO Postdoctoral Fellowship at Caltech. Prof. Anandkumar is supported by Bren endowed Chair, faculty awards from Microsoft, Google, and Adobe, and LwLL grants.
Funders:
Funding AgencyGrant Number
PIMCO Postdoctoral FellowshipUNSPECIFIED
Bren Professor of Computing and Mathematical SciencesUNSPECIFIED
Microsoft Faculty FellowshipUNSPECIFIED
Google Faculty Research AwardUNSPECIFIED
AdobeUNSPECIFIED
Learning with Less Labels (LwLL)UNSPECIFIED
Record Number:CaltechAUTHORS:20201110-074357009
Persistent URL:https://resolver.caltech.edu/CaltechAUTHORS:20201110-074357009
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:106577
Collection:CaltechAUTHORS
Deposited By: Tony Diaz
Deposited On:10 Nov 2020 16:22
Last Modified:10 Nov 2020 16:22

Repository Staff Only: item control page