A Caltech Library Service

Can You Label Less by Using Out-of-Domain Data? Active & Transfer Learning with Few-shot Instructions

Kocielnik, Rafal and Kangaslahti, Sara and Prabhumoye, Shrimai and Hari, Meena and Alvarez, R. Michael and Anandkumar, Anima (2022) Can You Label Less by Using Out-of-Domain Data? Active & Transfer Learning with Few-shot Instructions. . (Unpublished)

[img] PDF - Accepted Version
Creative Commons Attribution.


Use this Persistent URL to link to this item:


Labeling social-media data for custom dimensions of toxicity and social bias is challenging and labor-intensive. Existing transfer and active learning approaches meant to reduce annotation effort require fine-tuning, which suffers from over-fitting to noise and can cause domain shift with small sample sizes. In this work, we propose a novel Active Transfer Few-shot Instructions (ATF) approach which requires no fine-tuning. ATF leverages the internal linguistic knowledge of pre-trained language models (PLMs) to facilitate the transfer of information from existing pre-labeled datasets (source-domain task) with minimum labeling effort on unlabeled target data (target-domain task). Our strategy can yield positive transfer achieving a mean AUC gain of 10.5% compared to no transfer with a large 22b parameter PLM. We further show that annotation of just a few target-domain samples via active learning can be beneficial for transfer, but the impact diminishes with more annotation effort (26% drop in gain between 100 and 2000 annotated examples). Finally, we find that not all transfer scenarios yield a positive gain, which seems related to the PLMs initial performance on the target-domain task.

Item Type:Report or Paper (Discussion Paper)
Related URLs:
URLURL TypeDescription Paper
Kocielnik, Rafal0000-0001-5602-6056
Alvarez, R. Michael0000-0002-8113-4451
Anandkumar, Anima0000-0002-6974-6797
Additional Information:Attribution 4.0 International (CC BY 4.0). We would like to thank the Caltech SURF program for contributing to the funding of this project and especially the named donor Carolyn Ash. This material is based upon work supported by the National Science Foundation under Grant # 2030859 to the Computing Research Association for the CIFellows Project. Anima Anandkumar is partially supported by Bren Named Chair Professorship at Caltech and is a paid employee of Nvidia. Sara Kangaslahti was a paid part-time intern at Nvidia during this project.
Funding AgencyGrant Number
Caltech Summer Undergraduate Research Fellowship (SURF)UNSPECIFIED
Bren Professor of Computing and Mathematical SciencesUNSPECIFIED
Record Number:CaltechAUTHORS:20221221-004733367
Persistent URL:
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:118560
Deposited By: George Porter
Deposited On:22 Dec 2022 18:30
Last Modified:22 Dec 2022 18:30

Repository Staff Only: item control page