A Caltech Library Service

Social behavior recognition in continuous video

Burgos-Artizzu, Xavier P. and Dollár, Piotr and Lin, Dayu and Anderson, David J. and Perona, Pietro (2012) Social behavior recognition in continuous video. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE , Piscataway, NJ, pp. 1322-1329. ISBN 978-1-4673-1226-4.

Full text is not posted in this repository. Consult Related URLs below.

Use this Persistent URL to link to this item:


We present a novel method for analyzing social behavior. Continuous videos are segmented into action ‘bouts’ by building a temporal context model that combines features from spatio-temporal energy and agent trajectories. The method is tested on an unprecedented dataset of videos of interacting pairs of mice, which was collected as part of a state-of-the-art neurophysiological study of behavior. The dataset comprises over 88 hours (8 million frames) of annotated videos. We find that our novel trajectory features, used in a discriminative framework, are more informative than widely used spatio-temporal features; furthermore, temporal context plays an important role for action recognition in continuous videos. Our approach may be seen as a baseline method on this dataset, reaching a mean recognition rate of 61.2% compared to the expert's agreement rate of about 70%.

Item Type:Book Section
Related URLs:
URLURL TypeDescription
Anderson, David J.0000-0001-6175-3872
Perona, Pietro0000-0002-7583-5809
Additional Information:© 2012 IEEE. Date of Current Version: 26 July 2012. Authors would like to thank R. Robertson for his careful work annotating the videos, as well as Dr. A. Steele for coordinating some of the annotations. We also would like to thank Dr. M. Maire for his valuable feedback on the paper. X.P.B.A. holds a postdoctoral fellowship from the Spanish ministry of education, Programa Nacional de Movilidad de Recursos Humanos del Plan Nacional de I-D+i 2008-2011. D.L. was supported by the Jane Coffin Child Memorial Foundation. P.P. and D.J.A. were supported by the Gordon and Betty Moore Foundation. D.J. is supported by the Howard Hughes Medical foundation. P.P. was also supported by ONR MURI Grant #N00014-10-1-0933.
Funding AgencyGrant Number
Jane Coffin Childs Memorial Fund for Medical ResearchUNSPECIFIED
Gordon and Betty Moore FoundationUNSPECIFIED
Howard Hughes Medical Institute (HHMI)UNSPECIFIED
Office of Naval Research (ONR)N00014-10-1-0933
Record Number:CaltechAUTHORS:20120731-115125060
Persistent URL:
Official Citation:Burgos-Artizzu, Xavier P.; Dollar, Piotr; Lin, Dayu; Anderson, David J.; Perona, Pietro; , "Social behavior recognition in continuous video," Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on , vol., no., pp.1322-1329, 16-21 June 2012 doi: 10.1109/CVPR.2012.6247817 URL:
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:32817
Deposited By: Ruth Sustaita
Deposited On:01 Aug 2012 14:55
Last Modified:09 Nov 2021 21:30

Repository Staff Only: item control page