A Caltech Library Service

Task Programming: Learning Data Efficient Behavior Representations

Sun, Jennifer J. and Kennedy, Ann and Zhan, Eric and Yue, Yisong and Perona, Pietro (2020) Task Programming: Learning Data Efficient Behavior Representations. . (Unpublished)

[img] PDF - Submitted Version
See Usage Policy.


Use this Persistent URL to link to this item:


Specialized domain knowledge is often necessary to accurately annotate training sets for in-depth analysis, but can be burdensome and time-consuming to acquire from domain experts. This issue arises prominently in automated behavior analysis, in which agent movements or actions of interest are detected from video tracking data. To reduce annotation effort, we present TREBA: a method to learn annotation-sample efficient trajectory embedding for behavior analysis, based on multi-task self-supervised learning. The tasks in our method can be efficiently engineered by domain experts through a process we call "task programming", which uses programs to explicitly encode structured knowledge from domain experts. Total domain expert effort can be reduced by exchanging data annotation time for the construction of a small number of programmed tasks. We evaluate this trade-off using data from behavioral neuroscience, in which specialized domain knowledge is used to identify behaviors. We present experimental results in three datasets across two domains: mice and fruit flies. Using embeddings from TREBA, we reduce annotation burden by up to a factor of 10 without compromising accuracy compared to state-of-the-art features. Our results thus suggest that task programming can be an effective way to reduce annotation effort for domain experts.

Item Type:Report or Paper (Discussion Paper)
Related URLs:
URLURL TypeDescription Paper
Sun, Jennifer J.0000-0002-0906-6589
Kennedy, Ann0000-0002-3782-0518
Yue, Yisong0000-0001-9127-1989
Perona, Pietro0000-0002-7583-5809
Additional Information:We would like to thank David J. Anderson and the David Anderson Research Group at Caltech for this collaboration and the recording and annotation of the mouse behavior datasets, in particular, we would like to thank Tomomi Karigo for the behavior annotations. This work is partially supported by NIH Award #K99MH117264, NSF Award #1918839, and NSERC Award #PGSD3-532647-2019.
Funding AgencyGrant Number
Natural Sciences and Engineering Research Council of Canada (NSERC)PGSD3-532647-2019
Record Number:CaltechAUTHORS:20210119-161625521
Persistent URL:
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:107562
Deposited By: George Porter
Deposited On:20 Jan 2021 15:59
Last Modified:20 Jan 2021 15:59

Repository Staff Only: item control page