A Caltech Library Service

Automated measurement of mouse social behaviors using depth sensing, video tracking, and machine learning

Hong, Weizhe and Kennedy, Ann and Burgos-Artizzu, Xavier P. and Zelikowsky, Moriel and Navonne, Santiago G. and Perona, Pietro and Anderson, David J. (2015) Automated measurement of mouse social behaviors using depth sensing, video tracking, and machine learning. Proceedings of the National Academy of Sciences, 112 (38). E5351-E5360. ISSN 0027-8424. PMCID PMC4603510. doi:10.1073/pnas.1515982112.

[img] PDF - Published Version
See Usage Policy.

[img] PDF - Supplemental Material
See Usage Policy.

[img] Video (MPEG) (Movie S1) - Supplemental Material
See Usage Policy.

[img] Video (MPEG) (Movie S2) - Supplemental Material
See Usage Policy.

[img] Video (MPEG) (Movie S3) - Supplemental Material
See Usage Policy.

[img] Video (MPEG) (Movie S4) - Supplemental Material
See Usage Policy.


Use this Persistent URL to link to this item:


A lack of automated, quantitative, and accurate assessment of social behaviors in mammalian animal models has limited progress toward understanding mechanisms underlying social interactions and their disorders such as autism. Here we present a new integrated hardware and software system that combines video tracking, depth sensing, and machine learning for automatic detection and quantification of social behaviors involving close and dynamic interactions between two mice of different coat colors in their home cage. We designed a hardware setup that integrates traditional video cameras with a depth camera, developed computer vision tools to extract the body “pose” of individual animals in a social context, and used a supervised learning algorithm to classify several well-described social behaviors. We validated the robustness of the automated classifiers in various experimental settings and used them to examine how genetic background, such as that of Black and Tan Brachyury (BTBR) mice (a previously reported autism model), influences social behavior. Our integrated approach allows for rapid, automated measurement of social behaviors across diverse experimental designs and also affords the ability to develop new, objective behavioral metrics.

Item Type:Article
Related URLs:
URLURL TypeDescription Information CentralArticle
Hong, Weizhe0000-0003-1523-8575
Kennedy, Ann0000-0002-3782-0518
Burgos-Artizzu, Xavier P.0000-0001-9681-5404
Perona, Pietro0000-0002-7583-5809
Anderson, David J.0000-0001-6175-3872
Additional Information:© 2015 National Academy of Sciences. Contributed by David J. Anderson, August 16, 2015 (sent for review May 20, 2015). Published online before print September 9, 2015. We thank Xiao Wang and Xiaolin Da for manual video annotation; Michele Damian, Louise Naud, and Robert Robertson for assistance with coding; Allan Wong for helpful suggestions; Prof. Sandeep R. Datta (Harvard University) for sharing unpublished data; Celine Chiu for laboratory management; and Gina Mancuso for administrative assistance. This work was supported by grants from the Moore and Simons Foundations and postdoctoral support from the Helen Hay Whitney Foundation (W.H.), the National Science Foundation (M.Z.) and Sloan-Swartz Foundation (A.K.). D.J.A. is an Investigator of the Howard Hughes Medical Institute. Author contributions: W.H., P.P., and D.J.A. designed research; W.H. performed research; W.H., X.P.B.-A., and S.G.N. contributed new reagents/analytic tools; W.H., A.K., M.Z., P.P., and D.J.A. analyzed data; and W.H., A.K., M.Z., P.P., and D.J.A. wrote the paper. The authors declare no conflict of interest. This article contains supporting information online at
Funding AgencyGrant Number
Gordon and Betty Moore FoundationUNSPECIFIED
Simons FoundationUNSPECIFIED
Helen Hay Whitney FoundationUNSPECIFIED
Sloan-Swartz FoundationUNSPECIFIED
Howard Hughes Medical Institute (HHMI)UNSPECIFIED
Subject Keywords:social behavior; behavioral tracking; machine vision; depth sensing; supervised machine learning
Issue or Number:38
PubMed Central ID:PMC4603510
Record Number:CaltechAUTHORS:20150914-091025001
Persistent URL:
Official Citation:Weizhe Hong, Ann Kennedy, Xavier P. Burgos-Artizzu, Moriel Zelikowsky, Santiago G. Navonne, Pietro Perona, and David J. Anderson Automated measurement of mouse social behaviors using depth sensing, video tracking, and machine learning PNAS 2015 112 (38) E5351-E5360; published ahead of print September 9, 2015, doi:10.1073/pnas.1515982112
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:60222
Deposited By: Tony Diaz
Deposited On:14 Sep 2015 17:13
Last Modified:23 May 2022 17:13

Repository Staff Only: item control page