CaltechAUTHORS
  A Caltech Library Service

BKinD-3D: Self-Supervised 3D Keypoint Discovery from Multi-View Videos

Sun, Jennifer J. and Karashchuk, Pierre and Dravid, Amil and Ryou, Serim and Fereidooni, Sonia and Tuthill, John C. and Katsaggelos, Aggelos and Brunton, Bingni W. and Gkioxari, Georgia and Kennedy, Ann and Yue, Yisong and Perona, Pietro (2022) BKinD-3D: Self-Supervised 3D Keypoint Discovery from Multi-View Videos. . (Unpublished) https://resolver.caltech.edu/CaltechAUTHORS:20221219-204745839

[img] PDF - Submitted Version
See Usage Policy.

4MB

Use this Persistent URL to link to this item: https://resolver.caltech.edu/CaltechAUTHORS:20221219-204745839

Abstract

Quantifying motion in 3D is important for studying the behavior of humans and other animals, but manual pose annotations are expensive and time-consuming to obtain. Self-supervised keypoint discovery is a promising strategy for estimating 3D poses without annotations. However, current keypoint discovery approaches commonly process single 2D views and do not operate in the 3D space. We propose a new method to perform self-supervised keypoint discovery in 3D from multi-view videos of behaving agents, without any keypoint or bounding box supervision in 2D or 3D. Our method uses an encoder-decoder architecture with a 3D volumetric heatmap, trained to reconstruct spatiotemporal differences across multiple views, in addition to joint length constraints on a learned 3D skeleton of the subject. In this way, we discover keypoints without requiring manual supervision in videos of humans and rats, demonstrating the potential of 3D keypoint discovery for studying behavior.


Item Type:Report or Paper (Discussion Paper)
Related URLs:
URLURL TypeDescription
http://arxiv.org/abs/2212.07401arXivDiscussion Paper
ORCID:
AuthorORCID
Sun, Jennifer J.0000-0002-0906-6589
Karashchuk, Pierre0000-0001-6244-8239
Dravid, Amil0000-0001-6007-0690
Ryou, Serim0000-0003-1344-1158
Tuthill, John C.0000-0002-5689-5806
Katsaggelos, Aggelos0000-0003-4554-0070
Brunton, Bingni W.0000-0002-4831-3466
Kennedy, Ann0000-0002-3782-0518
Yue, Yisong0000-0001-9127-1989
Perona, Pietro0000-0002-7583-5809
Additional Information:This work is generously supported by the Amazon AI4Science Fellowship (to JJS), NIH NINDS (R01NS102333 to JCT), and the Air Force Office of Scientific Research (AFOSR FA9550-19-1-0386 to BWB).
Funders:
Funding AgencyGrant Number
Amazon AI4Science FellowshipUNSPECIFIED
NIHR01NS102333
Air Force Office of Scientific Research (AFOSR)FA9550-19-1-0386
Record Number:CaltechAUTHORS:20221219-204745839
Persistent URL:https://resolver.caltech.edu/CaltechAUTHORS:20221219-204745839
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:118408
Collection:CaltechAUTHORS
Deposited By: George Porter
Deposited On:20 Dec 2022 03:48
Last Modified:20 Dec 2022 03:48

Repository Staff Only: item control page