A Caltech Library Service

Task Programming: Learning Data Efficient Behavior Representations

Sun, Jennifer J. and Kennedy, Ann and Zhan, Eric and Anderson, David J. and Yue, Yisong and Perona, Pietro (2021) Task Programming: Learning Data Efficient Behavior Representations. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE , Piscataway, NJ, pp. 2875-2884. ISBN 978-1-6654-4509-2.

[img] PDF - Submitted Version
See Usage Policy.


Use this Persistent URL to link to this item:


Specialized domain knowledge is often necessary to accurately annotate training sets for in-depth analysis, but can be burdensome and time-consuming to acquire from domain experts. This issue arises prominently in automated behavior analysis, in which agent movements or actions of interest are detected from video tracking data. To reduce annotation effort, we present TREBA: a method to learn annotation-sample efficient trajectory embedding for behavior analysis, based on multi-task self-supervised learning. The tasks in our method can be efficiently engineered by domain experts through a process we call "task programming", which uses programs to explicitly encode structured knowledge from domain experts. Total domain expert effort can be reduced by exchanging data annotation time for the construction of a small number of programmed tasks. We evaluate this trade-off using data from behavioral neuroscience, in which specialized domain knowledge is used to identify behaviors. We present experimental results in three datasets across two domains: mice and fruit flies. Using embeddings from TREBA, we reduce annotation burden by up to a factor of 10 without compromising accuracy compared to state-of-the-art features. Our results thus suggest that task programming and self-supervision can be an effective way to reduce annotation effort for domain experts.

Item Type:Book Section
Related URLs:
URLURL TypeDescription Paper
Sun, Jennifer J.0000-0002-0906-6589
Kennedy, Ann0000-0002-3782-0518
Anderson, David J.0000-0001-6175-3872
Yue, Yisong0000-0001-9127-1989
Perona, Pietro0000-0002-7583-5809
Additional Information:© 2021 IEEE. We would like to thank Tomomi Karigo at Caltech for providing the mouse dataset. The Simons Foundation (Global Brain grant 543025 to PP) generously supported this work, and this work is partially supported by NIH Award #K99MH117264 (to AK), NSF Award #1918839 (to YY), and NSERC Award #PGSD3-532647-2019 (to JJS).
Group:Tianqiao and Chrissy Chen Institute for Neuroscience
Funding AgencyGrant Number
Simons Foundation543025
Natural Sciences and Engineering Research Council of Canada (NSERC)PGSD3-532647-2019
Record Number:CaltechAUTHORS:20210119-161625521
Persistent URL:
Official Citation:J. J. Sun, A. Kennedy, E. Zhan, D. J. Anderson, Y. Yue and P. Perona, "Task Programming: Learning Data Efficient Behavior Representations," 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 2875-2884, doi: 10.1109/CVPR46437.2021.00290
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:107562
Deposited By: George Porter
Deposited On:20 Jan 2021 15:59
Last Modified:10 Jan 2022 22:58

Repository Staff Only: item control page