A Caltech Library Service

Automated audiovisual behavior recognition in wild primates

Bain, Max and Nagrani, Arsha and Schofield, Daniel and Berdugo, Sophie and Bessa, Joana and Owen, Jake and Hockings, Kimberley J. and Matsuzawa, Tetsuro and Hayashi, Misato and Biro, Dora and Carvalho, Susana and Zisserman, Andrew (2021) Automated audiovisual behavior recognition in wild primates. Science Advances, 7 (46). Art. No. eabi4883. ISSN 2375-2548. PMCID PMC8589313. doi:10.1126/sciadv.abi4883.

[img] PDF - Published Version
Creative Commons Attribution Non-commercial.

[img] PDF (Supplementary Text; Figs. S1 to S3; Table S1; Legend for Movie S1) - Supplemental Material
Creative Commons Attribution Non-commercial.

[img] Archive (ZIP) (Movie S1) - Supplemental Material
Creative Commons Attribution Non-commercial.


Use this Persistent URL to link to this item:


Large video datasets of wild animal behavior are crucial to produce longitudinal research and accelerate conservation efforts; however, large-scale behavior analyses continue to be severely constrained by time and resources. We present a deep convolutional neural network approach and fully automated pipeline to detect and track two audiovisually distinctive actions in wild chimpanzees: buttress drumming and nut cracking. Using camera trap and direct video recordings, we train action recognition models using audio and visual signatures of both behaviors, attaining high average precision (buttress drumming: 0.87 and nut cracking: 0.85), and demonstrate the potential for behavioral analysis using the automatically parsed video. Our approach produces the first automated audiovisual action recognition of wild primate behavior, setting a milestone for exploiting large datasets in ethology and conservation.

Item Type:Article
Related URLs:
URLURL TypeDescription ItemData CentralArticle
Bain, Max0000-0002-2345-5441
Nagrani, Arsha0000-0003-2190-9013
Schofield, Daniel0000-0002-3308-0209
Berdugo, Sophie0000-0003-2162-1087
Bessa, Joana0000-0001-7196-5369
Owen, Jake0000-0002-8490-3949
Hockings, Kimberley J.0000-0002-6187-644X
Matsuzawa, Tetsuro0000-0002-8147-2725
Hayashi, Misato0000-0001-7289-6414
Biro, Dora0000-0002-3408-6274
Carvalho, Susana0000-0003-4542-3720
Zisserman, Andrew0000-0002-8945-8573
Additional Information:© 2021 The Authors, some rights reserved; exclusive licensee American Association for the Advancement of Science. No claim to original U.S. Government Works. Distributed under a Creative Commons Attribution NonCommercial License 4.0 (CC BY-NC). Submitted 12 March 2021; Accepted 23 September 2021; Published 12 November 2021. We are grateful to Kyoto University’s Primate Research Institute for leading the Bossou Archive Project and supporting the research presented here and to the IREB and DNRSIT of Guinea. This study is dedicated to all the researchers and field assistants who have collected data in Bossou since 1988. We thank the Instituto da Biodiversidade e das Áreas Protegidas (IBAP) for their permission to conduct research in Guinea-Bissau and for logistical support, research assistants and local guides for assisting with data collection, and local leaders for granting us permission to conduct research. We thank M. Ramon for collecting camera trap data in Cabante, Guinea-Bissau. This study was supported by EPSRC Programme Grants Seebibyte EP/M013774/1 and Visual AI EP/T028572/1; Google PhD Fellowship (to A.N.); Clarendon Fund (to D.S. and S.B.); Boise Trust Fund (to D.S., S.B., and J.B.); Wolfson College, University of Oxford (to D.S.); Keble College Sloane-Robinson Clarendon Scholarship, University of Oxford (to S.B.); Fundação para a Ciência e a Tecnologia, Portugal SFRH/BD/108185/2015 (to J.B.); Templeton World Charity Foundation grant no. TWCF0316 (to D.B.); National Geographic Society (to S.C.); St Hugh’s College, University of Oxford (to S.C.); Kyoto University Primate Research Institute for Cooperative Research Program (to M.H. and D.S.); MEXT-JSPS (no. 16H06283), LGP-U04, the Japan Society for the Promotion of Science (to T.M.); and Darwin Initiative funding grant number 26-018 (to K.J.H.). Author contributions: Conceptualization: D.S. Methodology: M.B., A.N., and A.Z. Data curation: M.B., D.S., J.B., S.B., and J.O. Data collection: D.B., S.C., T.M., M.H., K.J.H., and J.B. Software, formal analysis, and visualization: M.B. Supervision: A.Z., D.B., and S.C. Writing (original draft): M.B., A.N., D.S., and J.B. Writing (review and editing): A.Z., D.B., S.C., and K.J.H. The authors declare that they have no competing interests. Data and materials availability: All data needed to evaluate the conclusions in the paper are present in the paper and/or the Supplementary Materials and at the Dryad data repository (
Funding AgencyGrant Number
Engineering and Physical Sciences Research Council (EPSRC)EP/M013774/1
Engineering and Physical Sciences Research Council (EPSRC)EP/T028572/1
Google PhD FellowshipUNSPECIFIED
Wolfson College, OxfordUNSPECIFIED
Keble College Sloane-Robinson Clarendon Scholarship, OxfordUNSPECIFIED
Fundação para a Ciência e a Tecnologia (FCT)SFRH/BD/108185/2015
Templeton World Charity FoundationTWCF0316
National Geographic SocietyUNSPECIFIED
St Hugh’s College, OxfordUNSPECIFIED
Kyoto University Primate Research InstituteUNSPECIFIED
Ministry of Education, Culture, Sports, Science and Technology (MEXT)16H06283
Japan Society for the Promotion of Science (JSPS)LGP-U04
Darwin Initiative26-018
Issue or Number:46
PubMed Central ID:PMC8589313
Record Number:CaltechAUTHORS:20211115-173820409
Persistent URL:
Official Citation:Automated audiovisual behavior recognition in wild primates. Max Bain, Arsha Nagrani, Daniel Schofield, Sophie Berdugo, Joana Bessa, Jake Owen, Kimberley J. Hockings, Tetsuro Matsuzawa, Misato Hayashi, Dora Biro, Susana Carvalho, Andrew Zisserman. Sci. Adv., 7 (46), eabi4883; DOI: 10.1126/sciadv.abi4883
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:111868
Deposited By: Tony Diaz
Deposited On:15 Nov 2021 17:58
Last Modified:19 Nov 2021 17:07

Repository Staff Only: item control page