A Caltech Library Service

The Multi-Agent Behavior Dataset: Mouse Dyadic Social Interactions

Sun, Jennifer J. and Karigo, Tomomi and Chakraborty, Dipam and Mohanty, Sharada P. and Anderson, David J. and Perona, Pietro and Yue, Yisong and Kennedy, Ann (2021) The Multi-Agent Behavior Dataset: Mouse Dyadic Social Interactions. . (Unpublished)

PDF - Submitted Version
See Usage Policy.


Use this Persistent URL to link to this item:


Multi-agent behavior modeling aims to understand the interactions that occur between agents. We present a multi-agent dataset from behavioral neuroscience, the Caltech Mouse Social Interactions (CalMS21) Dataset. Our dataset consists of trajectory data of social interactions, recorded from videos of freely behaving mice in a standard resident-intruder assay. The CalMS21 dataset is part of the Multi-Agent Behavior Challenge 2021 and for our next step, our goal is to incorporate datasets from other domains studying multi-agent behavior. To help accelerate behavioral studies, the CalMS21 dataset provides a benchmark to evaluate the performance of automated behavior classification methods in three settings: (1) for training on large behavioral datasets all annotated by a single annotator, (2) for style transfer to learn inter-annotator differences in behavior definitions, and (3) for learning of new behaviors of interest given limited training data. The dataset consists of 6 million frames of unlabelled tracked poses of interacting mice, as well as over 1 million frames with tracked poses and corresponding frame-level behavior annotations. The challenge of our dataset is to be able to classify behaviors accurately using both labelled and unlabelled tracking data, as well as being able to generalize to new annotators and behaviors.

Item Type:Report or Paper (Discussion Paper)
Related URLs:
URLURL TypeDescription Paper ItemDataset and Challenge
Sun, Jennifer J.0000-0002-0906-6589
Anderson, David J.0000-0001-6175-3872
Perona, Pietro0000-0002-7583-5809
Yue, Yisong0000-0001-9127-1989
Kennedy, Ann0000-0002-3782-0518
Additional Information:We would like to thank the researchers at the David Anderson Research Group at Caltech for this collaboration and the recording and annotation of the mouse behavior datasets. We are grateful to the team at AICrowd for the support and hosting our dataset challenge, as well as Northwestern University and Amazon Sagemaker for funding our challenge prizes. This work was generously supported by the Simons Collaboration on the Global Brain grant 543025 (to PP), NIH Award #K99MH117264 (to AK), NSF Award #1918839 (to YY), and NSERC Award #PGSD3-532647-2019 (to JJS).
Group:Tianqiao and Chrissy Chen Institute for Neuroscience
Funding AgencyGrant Number
Simons Foundation543025
Natural Sciences and Engineering Research Council of Canada (NSERC)PGSD3-532647-2019
Record Number:CaltechAUTHORS:20210510-093610124
Persistent URL:
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:109027
Deposited By: Tony Diaz
Deposited On:10 May 2021 17:46
Last Modified:10 May 2021 17:46

Repository Staff Only: item control page