A Caltech Library Service

The MABe22 Benchmarks for Representation Learning of Multi-Agent Behavior

Sun, Jennifer J. and Ulmer, Andrew and Chakraborty, Dipam and Geuther, Brian and Hayes, Edward and Jia, Heng and Kumar, Vivek and Partridge, Zachary and Robie, Alice and Schretter, Catherine and Sun, Chao and Sheppard, Keith and Uttarwar, Param and Perona, Pietro and Yue, Yisong and Branson, Kristin and Kennedy, Ann (2022) The MABe22 Benchmarks for Representation Learning of Multi-Agent Behavior. . (Unpublished)

Full text is not posted in this repository. Consult Related URLs below.

Use this Persistent URL to link to this item:


Real-world behavior is often shaped by complex interactions between multiple agents. To scalably study multi-agent behavior, advances in unsupervised and self-supervised learning have enabled a variety of different behavioral representations to be learned from trajectory data. To date, there does not exist a unified set of benchmarks that can enable comparing methods quantitatively and systematically across a broad set of behavior analysis settings. We aim to address this by introducing a large-scale, multi-agent trajectory dataset from real-world behavioral neuroscience experiments that covers a range of behavior analysis tasks. Our dataset consists of trajectory data from common model organisms, with 9.6 million frames of mouse data and 4.4 million frames of fly data, in a variety of experimental settings, such as different strains, lengths of interaction, and optogenetic stimulation. A subset of the frames also consist of expert-annotated behavior labels. Improvements on our dataset corresponds to behavioral representations that work across multiple organisms and is able to capture differences for common behavior analysis tasks.

Item Type:Report or Paper (Discussion Paper)
Related URLs:
URLURL TypeDescription Paper
Sun, Jennifer J.0000-0002-0906-6589
Geuther, Brian0000-0002-7822-486X
Robie, Alice0000-0002-0784-2927
Schretter, Catherine0000-0002-3957-6838
Sheppard, Keith0000-0003-0842-9365
Perona, Pietro0000-0002-7583-5809
Yue, Yisong0000-0001-9127-1989
Branson, Kristin0000-0002-5567-2512
Kennedy, Ann0000-0002-3782-0518
Additional Information:This work was generously supported by the Simons Collaboration on the Global Brain grant 543025 (to PP), NIH Award #R00MH117264 (to AK), NSF Award #1918839 (to YY), NSERC Award #PGSD3-532647-2019 (to JJS), as well as a gift from Charles and Lily Trimble (to PP). We would like to thank Tom Sproule for mouse breeding and dataset collection. The mouse dataset was supported by the National Institute of Health DA041668 (NIDA), DA048634 (NIDA, and Simons Foundation SFARI Director’s Award) (to VK). We also greatly appreciate Google, Amazon, HHMI, and the Simons Foundation for sponsoring the MABe 2022 Challenge and Workshop.
Funding AgencyGrant Number
Simons Foundation543025
Natural Sciences and Engineering Research Council of Canada (NSERC)PGSD3-532647-2019
Charles and Lily TrimbleUNSPECIFIED
Record Number:CaltechAUTHORS:20221219-234042044
Persistent URL:
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:118462
Deposited By: George Porter
Deposited On:21 Dec 2022 01:03
Last Modified:02 Jun 2023 01:28

Repository Staff Only: item control page