A Caltech Library Service

The Mouse Action Recognition System (MARS): a software pipeline for automated analysis of social behaviors in mice

Segalin, Cristina and Williams, Jalani and Karigo, Tomomi and Hui, May and Zelikowsky, Moriel and Sun, Jennifer J. and Perona, Pietro and Anderson, David J. and Kennedy, Ann (2020) The Mouse Action Recognition System (MARS): a software pipeline for automated analysis of social behaviors in mice. . (Unpublished)

PDF - Submitted Version
See Usage Policy.

PDF (Supplemental Figures) - Supplemental Material
See Usage Policy.


Use this Persistent URL to link to this item:


The study of social behavior requires scoring the animals' interactions. This is generally done by hand-- a time consuming, subjective, and expensive process. Recent advances in computer vision enable tracking the pose (posture) of freely-behaving laboratory animals automatically. However, classifying complex social behaviors such as mounting and attack remains technically challenging. Furthermore, the extent to which expert annotators, possibly from different labs, agree on the definitions of these behaviors varies. There is a shortage in the neuroscience community of benchmark datasets that can be used to evaluate the performance and reliability of both pose estimation tools and manual and automated behavior scoring. We introduce the Mouse Action Recognition System (MARS), an automated pipeline for pose estimation and behavior quantification in pairs of freely behaving mice. We compare MARS's annotations to human annotations and find that MARS's pose estimation and behavior classification achieve human-level performance. As a by-product we characterize the inter-expert variability in behavior scoring. The two novel datasets used to train MARS were collected from ongoing experiments in social behavior, and identify the main sources of disagreement between annotators. They comprise 30,000 frames of manual annotated mouse poses and over 14 hours of manually annotated behavioral recordings in a variety of experimental preparations. We are releasing this dataset alongside MARS to serve as community benchmarks for pose and behavior systems. Finally, we introduce the Behavior Ensemble and Neural Trajectory Observatory (Bento), a graphical interface that allows users to quickly browse, annotate, and analyze datasets including behavior videos, pose estimates, behavior annotations, audio, and neural recording data. We demonstrate the utility of MARS and Bento in two use cases: a high-throughput behavioral phenotyping study, and exploration of a novel imaging dataset. Together, MARS and Bento provide an end-to-end pipeline for behavior data extraction and analysis, in a package that is user-friendly and easily modifiable.

Item Type:Report or Paper (Discussion Paper)
Related URLs:
URLURL TypeDescription Paper ItemData/Code
Segalin, Cristina0000-0001-7219-7074
Hui, May0000-0002-6231-7383
Zelikowsky, Moriel0000-0002-0465-9027
Sun, Jennifer J.0000-0002-0906-6589
Perona, Pietro0000-0002-7583-5809
Anderson, David J.0000-0001-6175-3872
Additional Information:The copyright holder for this preprint is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. Posted July 27, 2020.
Record Number:CaltechAUTHORS:20200728-092338077
Persistent URL:
Official Citation:The Mouse Action Recognition System (MARS): a software pipeline for automated analysis of social behaviors in mice. Cristina Segalin, Jalani Williams, Tomomi Karigo, May Hui, Moriel Zelikowsky, Jennifer J Sun, Pietro Perona, David J Anderson, Ann Kennedy. bioRxiv 2020.07.26.222299; doi:
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:104600
Deposited By: Tony Diaz
Deposited On:28 Jul 2020 17:23
Last Modified:28 Jul 2020 17:23

Repository Staff Only: item control page