Bain, Max and Nagrani, Arsha and Schofield, Daniel and Berdugo, Sophie and Bessa, Joana and Owen, Jake and Hockings, Kimberley J. and Matsuzawa, Tetsuro and Hayashi, Misato and Biro, Dora and Carvalho, Susana and Zisserman, Andrew (2021) Automated audiovisual behavior recognition in wild primates. Science Advances, 7 (46). Art. No. eabi4883. ISSN 2375-2548. PMCID PMC8589313. doi:10.1126/sciadv.abi4883. https://resolver.caltech.edu/CaltechAUTHORS:20211115-173820409
![[img]](https://authors.library.caltech.edu/style/images/fileicons/application_pdf.png) |
PDF
- Published Version
Creative Commons Attribution Non-commercial. 3MB |
|
PDF (Supplementary Text; Figs. S1 to S3; Table S1; Legend for Movie S1)
- Supplemental Material
Creative Commons Attribution Non-commercial. 6MB |
![[img]](https://authors.library.caltech.edu/style/images/fileicons/application_zip.png) |
Archive (ZIP) (Movie S1)
- Supplemental Material
Creative Commons Attribution Non-commercial. 33MB |
Use this Persistent URL to link to this item: https://resolver.caltech.edu/CaltechAUTHORS:20211115-173820409
Abstract
Large video datasets of wild animal behavior are crucial to produce longitudinal research and accelerate conservation efforts; however, large-scale behavior analyses continue to be severely constrained by time and resources. We present a deep convolutional neural network approach and fully automated pipeline to detect and track two audiovisually distinctive actions in wild chimpanzees: buttress drumming and nut cracking. Using camera trap and direct video recordings, we train action recognition models using audio and visual signatures of both behaviors, attaining high average precision (buttress drumming: 0.87 and nut cracking: 0.85), and demonstrate the potential for behavioral analysis using the automatically parsed video. Our approach produces the first automated audiovisual action recognition of wild primate behavior, setting a milestone for exploiting large datasets in ethology and conservation.
| Item Type: | Article | ||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Related URLs: |
| ||||||||||||||||||||||||||||||||
| ORCID: |
| ||||||||||||||||||||||||||||||||
| Additional Information: | © 2021 The Authors, some rights reserved; exclusive licensee American Association for the Advancement of Science. No claim to original U.S. Government Works. Distributed under a Creative Commons Attribution NonCommercial License 4.0 (CC BY-NC). Submitted 12 March 2021; Accepted 23 September 2021; Published 12 November 2021. We are grateful to Kyoto University’s Primate Research Institute for leading the Bossou Archive Project and supporting the research presented here and to the IREB and DNRSIT of Guinea. This study is dedicated to all the researchers and field assistants who have collected data in Bossou since 1988. We thank the Instituto da Biodiversidade e das Áreas Protegidas (IBAP) for their permission to conduct research in Guinea-Bissau and for logistical support, research assistants and local guides for assisting with data collection, and local leaders for granting us permission to conduct research. We thank M. Ramon for collecting camera trap data in Cabante, Guinea-Bissau. This study was supported by EPSRC Programme Grants Seebibyte EP/M013774/1 and Visual AI EP/T028572/1; Google PhD Fellowship (to A.N.); Clarendon Fund (to D.S. and S.B.); Boise Trust Fund (to D.S., S.B., and J.B.); Wolfson College, University of Oxford (to D.S.); Keble College Sloane-Robinson Clarendon Scholarship, University of Oxford (to S.B.); Fundação para a Ciência e a Tecnologia, Portugal SFRH/BD/108185/2015 (to J.B.); Templeton World Charity Foundation grant no. TWCF0316 (to D.B.); National Geographic Society (to S.C.); St Hugh’s College, University of Oxford (to S.C.); Kyoto University Primate Research Institute for Cooperative Research Program (to M.H. and D.S.); MEXT-JSPS (no. 16H06283), LGP-U04, the Japan Society for the Promotion of Science (to T.M.); and Darwin Initiative funding grant number 26-018 (to K.J.H.). Author contributions: Conceptualization: D.S. Methodology: M.B., A.N., and A.Z. Data curation: M.B., D.S., J.B., S.B., and J.O. Data collection: D.B., S.C., T.M., M.H., K.J.H., and J.B. Software, formal analysis, and visualization: M.B. Supervision: A.Z., D.B., and S.C. Writing (original draft): M.B., A.N., D.S., and J.B. Writing (review and editing): A.Z., D.B., S.C., and K.J.H. The authors declare that they have no competing interests. Data and materials availability: All data needed to evaluate the conclusions in the paper are present in the paper and/or the Supplementary Materials and at the Dryad data repository (https://datadryad.org/stash/share/UUfSTzsL9eTbAo-78pdaXPdaIUJmdJzSuqhXcb48vHM). | ||||||||||||||||||||||||||||||||
| Funders: |
| ||||||||||||||||||||||||||||||||
| Issue or Number: | 46 | ||||||||||||||||||||||||||||||||
| PubMed Central ID: | PMC8589313 | ||||||||||||||||||||||||||||||||
| DOI: | 10.1126/sciadv.abi4883 | ||||||||||||||||||||||||||||||||
| Record Number: | CaltechAUTHORS:20211115-173820409 | ||||||||||||||||||||||||||||||||
| Persistent URL: | https://resolver.caltech.edu/CaltechAUTHORS:20211115-173820409 | ||||||||||||||||||||||||||||||||
| Official Citation: | Automated audiovisual behavior recognition in wild primates. Max Bain, Arsha Nagrani, Daniel Schofield, Sophie Berdugo, Joana Bessa, Jake Owen, Kimberley J. Hockings, Tetsuro Matsuzawa, Misato Hayashi, Dora Biro, Susana Carvalho, Andrew Zisserman. Sci. Adv., 7 (46), eabi4883; DOI: 10.1126/sciadv.abi4883 | ||||||||||||||||||||||||||||||||
| Usage Policy: | No commercial reproduction, distribution, display or performance rights in this work are provided. | ||||||||||||||||||||||||||||||||
| ID Code: | 111868 | ||||||||||||||||||||||||||||||||
| Collection: | CaltechAUTHORS | ||||||||||||||||||||||||||||||||
| Deposited By: | Tony Diaz | ||||||||||||||||||||||||||||||||
| Deposited On: | 15 Nov 2021 17:58 | ||||||||||||||||||||||||||||||||
| Last Modified: | 19 Nov 2021 17:07 |
Repository Staff Only: item control page
