A Caltech Library Service

Self-Supervised Keypoint Discovery in Behavioral Videos

Sun, Jennifer J. and Ryou, Serim and Goldshmid, Roni and Weissbourd, Brandon and Dabiri, John and Anderson, David J. and Kennedy, Ann and Yue, Yisong and Perona, Pietro (2021) Self-Supervised Keypoint Discovery in Behavioral Videos. . (Unpublished)

[img] PDF - Submitted Version
See Usage Policy.


Use this Persistent URL to link to this item:


We propose a method for learning the posture and structure of agents from unlabelled behavioral videos. Starting from the observation that behaving agents are generally the main sources of movement in behavioral videos, our method uses an encoder-decoder architecture with a geometric bottleneck to reconstruct the difference between video frames. By focusing only on regions of movement, our approach works directly on input videos without requiring manual annotations, such as keypoints or bounding boxes. Experiments on a variety of agent types (mouse, fly, human, jellyfish, and trees) demonstrate the generality of our approach and reveal that our discovered keypoints represent semantically meaningful body parts, which achieve state-of-the-art performance on keypoint regression among self-supervised methods. Additionally, our discovered keypoints achieve comparable performance to supervised keypoints on downstream tasks, such as behavior classification, suggesting that our method can dramatically reduce the cost of model training vis-a-vis supervised methods.

Item Type:Report or Paper (Discussion Paper)
Related URLs:
URLURL TypeDescription Paper
Sun, Jennifer J.0000-0002-0906-6589
Goldshmid, Roni0000-0001-9095-3259
Weissbourd, Brandon0000-0001-5422-3873
Dabiri, John0000-0002-6722-9008
Anderson, David J.0000-0001-6175-3872
Kennedy, Ann0000-0002-3782-0518
Yue, Yisong0000-0001-9127-1989
Perona, Pietro0000-0002-7583-5809
Additional Information:This work was generously supported by the Simons Collaboration on the Global Brain grant 543025 (to PP and DJA), NIH Award #R00MH117264 (to AK), NSF Award #1918839 (to YY), NINDS Award #K99NS119749 (to BW), NIH Award #R01MH123612 (to DJA, PP, and SR), NSERC Award #PGSD3-532647-2019 (to JJS), as well as a gift from Charles and Lily Trimble (to PP).
Group:Tianqiao and Chrissy Chen Institute for Neuroscience, GALCIT
Funding AgencyGrant Number
Simons Foundation543025
Natural Sciences and Engineering Research Council of Canada (NSERC)PGSD3-532647-2019
Charles and Lily TrimbleUNSPECIFIED
Record Number:CaltechAUTHORS:20220224-200833645
Persistent URL:
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:113586
Deposited By: George Porter
Deposited On:28 Feb 2022 15:32
Last Modified:28 Feb 2022 15:32

Repository Staff Only: item control page