nihms97287.pdf

Automated Monitoring and Analysis of Social Behavior in

Drosophila

Heiko Dankert

1,2

Liming Wang

Eric D. Hoopfer

David J. Anderson

, and

Pietro Perona

Division of Engineering and Applied Science, California Institute of Technology, 1200 E California Blvd.,

Pasadena, CA 91125, USA

Division of Biology, Howard Hughes Medical Institute, California Institute of Technology, 1200 E

California Blvd., Pasadena, CA 91125, USA

Abstract

We introduce a method based on machine vision for automatically measuring aggression and

courtship in

Drosophila melanogaster

. The genetic and neural circuit bases of these innate social

behaviors are poorly understood. High-throughput behavioral screening in this genetically tractable

model organism is a potentially powerful approach, but it is currently very laborious. Our system

monitors interacting pairs of flies, and computes their location, orientation and wing posture. These

features are used for detecting behaviors exhibited during aggression and courtship. Among these,

wing threat, lunging and tussling are specific to aggression; circling, wing extension (courtship

“song”) and copulation are specific to courtship; locomotion and chasing are common to both.

Ethograms may be constructed automatically from these measurements, saving considerable time

and effort. This technology should enable large-scale screens for genes and neural circuits controlling

courtship and aggression.

INTRODUCTION

How are innate behaviors programmed into the genome? Answering this question requires

identifying the genes that control specific behaviors, the neural circuitry on which they act,

and how this circuitry controls behavior

1–5

. This may be attempted in model organisms such

as the nematode

Caenorhabditis elegans

and the fruit fly

Drosophila melanogaster

, thanks to

the abundant genetically based tools for marking, mapping and manipulating specific

populations of neurons

6,7

, thereby enabling large-scale genetic and functional screens

8,9

Social behaviors, such as courtship and aggression, are of particular interest, because they have

strong innate components. Mating has been studied in both

Drosophila

and

C. elegans

using

a combination of molecular genetic and cellular approaches

5,10

Drosophila

is unique amongst

invertebrate genetic model organisms in that it exhibits both courtship and aggression

11–14

Measuring animal behavior is difficult. Both aggression and courtship consist of rich ensembles

of stereotyped behaviors, which often unfold in a characteristic sequence

13,15

. Currently these

behaviors are measured manually, which is slow and laborious. Subjective decisions by the

observer may lead to difficulty in reproducing experiments. Furthermore, human observers

may fail to detect behavioral events that are too quick or too slow, and may miss events due

Corresponding Authors: D.J. Anderson, (626) 395-6821, E-mail: wuwei@caltech.edu; P. Perona, (626) 395-4867, E-mail:

perona@caltech.edu.

NIH Public Access

Author Manuscript

Nat Methods

. Author manuscript; available in PMC 2009 October 1.

Published in final edited form as:

Nat Methods

. 2009 April ; 6(4): 297–303. doi:10.1038/nmeth.1310.

NIH-PA Author Manuscript

to flagging attention. These constitute substantial obstacles to conducting large-scale

behavioral screens.

These limitations could be overcome through automation. The first step towards measuring

behavior is tracking (i.e., measuring the position and orientation of animals over time). Machine

vision systems have been designed for tracking houseflies

, mice

, ants

, bees

, single

Drosophila

in 3D

and for measuring

Drosophila

locomotion

9,21,22

showing promising

accuracy and flexibility. However, we do not yet have systems for measuring complex

behaviors automatically. A machine vision apparatus that detects lunging, an aggressive

behavior in

Drosophila,

has been developed recently

. In addition to lunging, it would be

desirable to measure other aggressive behaviors, such as chasing, tussling, boxing (fencing)

and wing threat

, as well as a suite of courtship behaviors. This would allow the study of

whether a given mutation or environmental influence exerts a selective effect on aggression,

or on social interactions in general

Here we describe a machine vision system designed to quantify and analyze a broad spectrum

of behaviors associated with both aggression and courtship in

Drosophila

. We designed our

system, first, to allow detailed, accurate and reproducible quantitative measurements of

individual component behaviors (“actions”) that are expressed in courtship and/or aggression

(“activities”); and second, to enable large-scale genetic and circuit-perturbation screens. The

system is simple and inexpensive to build and replicate, functions automatically and permits

measurements of multiple fly pairs simultaneously. The application of this approach, together

with appropriate multiplex aggression arena configurations

(see also Supplementary

Methods online), should enable large-scale genetic and circuit-based screens of these

behaviors.

RESULTS

We used a double-arena adapted from a recently developed system

(Fig. 1a). A consumer

camcorder, connected to a PC, filmed one pair of flies per arena. This setup can be adapted to

a medium-throughput mode, consisting of four double-arenas (Fig. 1b–d). Additional details

in the Supplementary Methods.

Our software consists of six modules: video import, ground-truthing, calibration, fly detection

and tracking, action detection and graphical user interface. Fly detection and tracking, as well

as action detection, are described here and in Figure 2, for other modules see Supplementary

Methods.

Detection and tracking

The first step in tracking the flies is computing their silhouette (body and wings) (Fig. 2a–c).

An important and novel feature of our software is the ability to detect and measure the position

of the fly’s head and wings, as illustrated in Figure 2c,d. Computing the orientation

of a fly

is described in Figure 2e. The bodies of abutting flies are resolved by fitting a two-component

Gaussian mixture model

simultaneously to pixel location and brightness (Fig. 2f). For details

see Supplementary Methods.

At each video frame 25 measurements are computed, characterizing body size, wing pose, and

position and velocity of the fly pair (Fig. 2d, e, g and Supplementary Table 1 online). These

measurements are the features used to detect actions.

In male-female assays the flies’ identity is directly measurable since the female is larger. In

male-male assays a white dot is painted on the back of one fly to allow individual identification.

Dankert et al.

Page 2

Nat Methods

. Author manuscript; available in PMC 2009 October 1.

NIH-PA Author Manuscript

For unlabeled male-male pairs our software computes the most likely fly-specific trajectories

(see Supplementary Methods).

Action detection

The twenty five features defined in the previous section are used for detecting fly actions.

Lunging, tussling and wing threat are actions specific to aggression; wing extension

(courtship “song”),

circling

and copulation are specific to courtship; locomotion and chasing

are common to both.

A lunge is defined as one fly rearing briefly on its hind legs and snapping down onto the other

fly

(Fig. 3a and Supplementary Fig. 1a online). Lunging is detected automatically by an

example-based classifier in a two-step process (Supplementary Methods for details). First,

probable lunges are selected amongst all frames by using ranges (intervals) on feature values

as listed in Supplementary Table 2 online. Second, probable lunges are accepted/rejected by a

k-nearest-neighbor classifier

using 10 features (see Supplementary Table 3 online). The

training examples consist of

≈

250 distinct expert-selected lunge events occurring over 8 fly

pairs × 20 minutes of recorded video, and of a comparable number of negative examples.

We evaluated the performance of our lunge detector on a 20-minute movie containing a large

number of lunges (139). Ground-truth, i.e. the accurate identification of all lunges in a movie,

was obtained following a two-step process (see Materials and Methods and Supplementary

Methods). We compared both the algorithm’s performance, as well as a second expert’s

annotations, to ground truth. Figure 4a shows the receiver operating characteristic

(ROC)

representing the fraction of false-negatives (lunges present in the ground truth, but not detected)

vs the number of false-positives (detected, but not present in the ground truth), as the number

of lunge-labeled nearest neighbors that are necessary to declare a lunge is varied. The threshold

we selected for labeling a lunge yields a detection rate of

≈

91%, representing 126 correct

detections, with 13 false-negatives, and 7 false-positives (encircled). Lowering the detection

threshold decreases the probability for false-negatives and increases the number of false-

positives.

We applied the lunge detector to 56 additional fly pair movies (Fig. 4b) and found excellent

agreement with the ground truth, with a correlation of 0.99, a bias of 1.5 lunges and a standard

error of the mean of 0.40 lunges.

The detection of other aggressive and courtship actions could be achieved by using only the

first of the two steps in the example-based classifier algorithm described above. Features and

ranges for each action were determined empirically by expert analysis of movies containing

sample actions. The k-nearest-neighbor classifier step was unnecessary in these cases.

In aggressive tussling, both flies grip each other with their front legs

(Fig. 3b and

Supplementary Fig. 1a online). The bodies face each other so that their axes of symmetry are

parallel (body alignment) and form a single line. Connected solidly in this configuration, they

move about in jerks at high velocity and acceleration. In order to detect and classify aggressive

tussling, 8 features are compared to ranges as listed in Supplementary Table 4 online. When

all features are within their empirically determined ranges and this configuration is maintained

for 0.3 s or longer we flag an aggressive tussling event.

Wing threat is characterized by a lateral extension of both wings by 80°–90° followed by their

elevation to a vertical extension of

≈

40° (Fig. 3c and Supplementary Fig. 1a online). We

observe both rapid, transient elevation of the wings, as well as longer-lasting (

≥

0.3 s)

occurrences; the latter are typically, but not always, associated with a reduction in walking

speed (velocity

≤

5 mm/s). In this study we have restricted detection of “wing threat” to longer-

Dankert et al.

Page 3

Nat Methods

. Author manuscript; available in PMC 2009 October 1.

NIH-PA Author Manuscript

lasting occurrences, as they are more easily discriminated from rapid “wing flicking” (see

Supplementary Table 5 online for features and ranges).

During courtship, males extend their wings laterally and vibrate them (

≈

280Hz) to produce a

“courtship song” (Fig. 3e and Supplementary Fig. 1b online). Wing extension is detected when

the angle between the wing and the long axis of the fly body is greater than 60°, the fly is not

standing up (fly length, as viewed from the camera, is maximal), and this configuration is

maintained for 1 s or longer (see Supplementary Table 6 online for features and ranges).

Circling is a behavioral action that is part of courtship, and is detected when one fly drifts

sideways in a circle with approximately constant velocity around the other fly (Fig. 3e and

Supplementary Fig. 1b online). Supplementary Table 7 online lists features and ranges.

Copulation involves a male fly approaching and mounting a female fly (Fig. 3d and

Supplementary Fig. 1b online). The beginning and end of copulation are characterized by an

abrupt change in the distance between the two flies. During copulation their movements

become coupled and locomotion is decreased. Thus, to detect both the starting and ending time-

points of copulation, we compute the mean and standard deviation of inter-fly distances

within a moving 250-frame (8.3 s) window. The earliest frame when the criteria of mean

distance < 2 mm and standard deviation < 0.3 mm are simultaneously met is defined as the

“copulation start”. The last such frame is defined as the “copulation end”. Supplementary Table

8 online lists features and ranges.

Chasing is detected when the change of the head-center distance

δΔ

h-c

between both flies (see

Fig. 2g) is small, both flies have the same, constant velocity, the distance between the flies is

small but not zero, the chasing fly is oriented towards the chased fly, and the head of the chasing

fly is behind the chased fly’s abdomen (Fig. 3f and Supplementary Fig. 1c online). This

configuration has to be maintained for 1 s or longer. Supplementary Table 9 online lists features

and ranges.

The performance of our system in detecting the actions described above was evaluated using

the same approach as in the case of lunging. For all of these behaviors we measured detection

rates between 90% and 100% (Table 1).

Automated analysis of genetic and environmental influences on aggressiveness and

courtship

To validate the utility of our system for studying experimental perturbations of courtship and

aggressive behavior, we first investigated whether it could detect previously described

phenotypes produced by gene- or circuit-level manipulations. Octopamine (OA) is an insect

biogenic amine, which is closely related to mammalian noradrenaline. It plays a critical role

in aggressive behavior in

Drosophila

. A recent study showed that silencing of OA neurons

decreases aggressive behavior

. We performed a similar manipulation, by expressing the

inwardly rectifying potassium channel Kir2.1

in tyrosine decarboxylase-2 (Tdc2)-

expressing neurons in order to suppress their electrical activity.

Tdc2-Gal4; UAS-Kir2.1

flies

showed significant decreases in lunging, tussling and wing-threats (Fig. 5a–d, Supplementary

Fig. 2a and 3a,b online). There was no statistically significant change in chasing, or total

distance traveled (Supplementary Fig. 2b,c online).

We also examined flies bearing a mutation in

fruitless

(Fru), a sex-specifically spliced

transcription factor that specifies gender-dimorphic fly behaviors

2,4

. Male Fru

flies, in which

the

fruitless

gene is spliced into a female-specific (inactive) form, exhibited a strong reduction

in male-specific patterns of aggressive behavior (Supplementary Fig. 4a–d online), as

previously reported

Dankert et al.

Page 4

Nat Methods

. Author manuscript; available in PMC 2009 October 1.

NIH-PA Author Manuscript

Further, we studied the behavior of

Cha-Gal4;UAS-tra

(“Cha-Tra”) flies, in which all

cholinergic neurons have been feminized due to mis-expression of the

transformer

gene

Cha-Tra males exhibited little or no courtship activity towards females, but a robust increase

in courtship towards other males (Fig. 5e,f and Supplemental Fig. 2d online), at a frequency

indistinguishable from wild-type Canton-S (CS) male-female pairs.

Cha-Tra male pairs also showed an increase in some aggressive behaviors compared to CS

male pairs (Fig. 5g,h, Supplementary Fig. 2e and 3c,d online) and greater locomotor activity

(Supplementary Fig. 2f online). Lunging activity was significantly higher than in controls, even

after normalizing for distance traveled

(Fig. 5i). Surprisingly, wing threat was significantly

lower in Cha-Tra flies (Fig. 5j); thus, not all aggressive actions were more frequent in Cha-Tra

flies. Nevertheless, the total time spent in aggressive activity (Supplementary Fig. 2g online)

and chasing (Supplementary Fig. 2h online) was significantly elevated in Cha-Tra males.

Control experiments indicated that copulation could be detected in CS male-female pairs

(Supplementary Fig. 2i online).

Our software allows us to compute 2D histograms showing the frequency of actions in each

spatial location in order to detect phenotypes with an altered spatial distribution of behaviors

(Supplementary Fig. 5 online). The 2D histograms revealed that Cha-Tra males performed a

greater proportion of their tussling bouts on the central food patch, in comparison to controls

(Supplementary Fig. 5a online). Moreover, the pattern of Cha-Tra chasing was more intense

around the perimeter of the arena, while that in controls was more uniformly distributed

(Supplementary Fig. 5b online).

We examined ethograms

, illustrating the frequency of each action, as well as the frequency

with which one action was followed by the same or another action. Both CS and Cha-Tra males

exhibited multiple transitions between courtship and aggressive activities (Fig. 5k–m). Cha-

Tra males additionally show transitions between aggression and courtship and vice versa. They

also show an elevated amount of chasing activity. In CS male pairs, chasing was followed most

often by lunging, whereas in Cha-Tra males, chasing was followed with equal probability by

either lunging, an aggressive action, or by wing extension, a courtship action (Figs. 5k and 5m,

light blue circles). By contrast, in CS male-female pairs, chasing by males was followed most

often by wing-extension (Fig. 5l). One interpretation is that in CS male-male pairs, chasing is

primarily an aggressive action, while in male-female pairs it is primarily a courtship action. In

Cha-Tra male-male pairs, chasing may be indicative of either aggression or courtship,

suggesting that these flies may have a deficit in gender recognition or discrimination. This

observation is consistent with a recent study

reporting that feminization of octopaminergic/

tyraminergic neurons by misexpression of

Tra

caused male-directed wing extensions to be

followed primarily by male-male courtship.

Finally, to examine the effects of social experience, an environmental influence, on male-

female courtship, we used our software to analyze wing-extension and circling, two male-

specific courtship actions. The male was in the presence of a decapitated female in the center

of the arena. Previous studies have shown that raising post-eclosion

Drosophila

males in social

isolation strongly increases aggressiveness, in comparison to flies raised in groups

. We

observed that both wing-extension and circling were significantly elevated in socially isolated

(SS) vs. group-housed (GG) flies (Fig. 5n,o). Thus, social isolation increases both male-male

aggressiveness and male-female courtship. This poses the question of whether these influences

on social behavior are mediated by common or distinct mechanisms.

Dankert et al.

Page 5

Nat Methods

. Author manuscript; available in PMC 2009 October 1.

NIH-PA Author Manuscript

DISCUSSION

We describe a method for measuring automatically a broad array of social behaviors in

Drosophila

. Actions associated with courtship, aggression and locomotion are detected from

overhead videos of fly pairs. We evaluated the system’s performance by comparing its

measurements with carefully established ground truth. For all actions we found detection rates

of 90–100% and comparably small false detection rates (see Table 1).

Our software detects wing postures, permitting measurements of wing threat and wing-

extension. Wing threat, in particular, is an interesting and important aggressive display because

it is independent of locomotor activity. Indeed, we show that a genetic manipulation (Cha-Tra)

that strongly increases lunging, tussling and chasing actually decreases wing threat.

The time saved by our software is enormous. In our experience, it takes at least one hour to

score manually one type of action in one 20 minute fly-pair movie. If one wished to characterize

aggression and courtship in a line of flies (all eight actions in, say, twelve fly pairs), one would

have to spend

≈

100 hours of manual labor, as opposed to a few minutes to set up and run our

system. As an example, the three ethograms presented in Figure 5 would have taken

≈

270

person-hours to prepare. This capability affords the opportunity to compare multiple genotypes

or wild-type genetic backgrounds, which would be virtually impossible to do manually.

The ability to monitor simultaneously both aggressive and courtship activities allows the

computation of ethograms. This may prove to be valuable in determining whether aggression

and courtship actions are part of a social behavior “continuum”, or whether these two activities

represent discrete “states” with “state transitions”, controlled by different circuits. Genetic or

circuit-based screens can be performed to search for phenotypes that affect not only the ability

to perform a particular action, but also that affect the probabilities of transition between actions.

Our system (see Supplementary Methods and Software online) is completely automatic and

self-calibrating; using it does not require special training, provided that the hardware setup is

well reproduced. It has been designed for inexpensive implementation and easy replication.

We found that fly behavior may be measured accurately in smaller arenas, allowing us to

simultaneously monitor an array of arenas with each camera, thus permitting large-scale

genetic screens with high throughput. Together, these features should open up aggression and

courtship to powerful genetic screens. This in turn should help to illuminate the genes and

neural circuits that control these important social behaviors, and may reveal general principles

of the organizational logic or control mechanisms that are evolutionarily conserved.

MATERIALS AND METHODS

Fly stocks and rearing conditions

Flies were reared in plastic vials containing standard fly medium (yeast, corn syrup, agar), at

25°C, 60% humidity with a 12 hours light-dark cycle. Newly eclosed males were single housed

or group housed (10 flies per vial) for 4–6 days before performing the behavioral assay. Virgin

Canton-S (CS) females were collected shortly after eclosion and raised at 20 flies per vial for

4–6 days before the courtship assay.

Cha-Gal4;UAS-tra

flies were made by crossing male

UAS-transformer to female

Cha-Gal4

flies. Fru

mutant was generated in B. Dickson lab.

Tdc2Gal4:UAS-Kir2.1

flies were made by crossing male

Tdc2-Gal4

flies to female

UAS-

Kir2.1

flies.

Dankert et al.

Page 6

Nat Methods

. Author manuscript; available in PMC 2009 October 1.

NIH-PA Author Manuscript

Aggression assay

We introduced two males of the same age into the double-arena setup (Fig. 1) by gentle

aspiration without anesthesia and immediately video-captured them for 20 minutes. The

temperature and humidity of the apparatus were set to

≈

25°C and

≈

40%, respectively.

Courtship assay

Two types of courtship assays were performed. The apparatus and environmental conditions

were as used in the aggression assay. One male and a virgin female were introduced into the

apparatus by gentle aspiration without anesthesia and immediately video-captured for 30

minutes (24 pairs) to cover the copulation period. For all other actions only the first 20 minutes

were analyzed. In another assay the female was decapitated and placed in the center of the food

patch and replaced every hour. After the male was introduced into the apparatus both flies were

immediately video-captured for 10 minutes.

Training and ground-truth data

We collected a hand-annotated database of lunging, wing threat, chasing, wing extension, and

circling to train our software and to measure its performance. Data used to train the detectors

were produced by a human observer without further checks. Data that were used for testing

the system’s performance were further processed to obtain a reliable ‘ground truth’. Human

experts tend to miss relevant events (in our observations, between 30% and 40% of events are

missed) for two reasons: (a) Different human observers will use slightly different criteria, even

when they agree on the overall action definition. (b) A human observer’s attention level will

change over time during movie annotations. Therefore, in order to obtain reliable ground truth

for performance testing, we devised an improved two-step procedure involving two experts

(see Supplementary Methods).

Statistical Analyses

Kruskal-Wallis-ANOVA was applied to detect overall differences among the unpaired groups.

Significantly different groups were compared pairwise by the Mann-Whitney U-test. In all

figures with statistics, one, two, and three asterisks indicate an

-level of 0.05, 0.01, and 0.001,

respectively. For all multiple comparisons, Bonferroni correction was applied.

Graphical user interfaces

Our software (Supplementary Fig. 6a,b and Software online) was run from graphical user

interfaces allowing the user to visualize tracking and statistical data.

Additional methods

In Supplementary Methods online we provide further information on the software, detection

and tracking, as well as method hardware. An executable of the software is provided in the

Supplementary Software. A free license of the source code is available on request to academic

and non-profit investigators. Commercial entities should contact the Caltech Office of

Technology Transfer for licensing arrangements.

Supplementary Material

Refer to Web version on PubMed Central for supplementary material.

Acknowledgments

We thank K. Watanabe and A. Hergarden for preparing the flies, assays, and taking video footage of flies as well as

helping with the ground-truth data. This work was supported by a NSF FIBR grant to M.J. Dickinson, D.J. Anderson

Dankert et al.

Page 7

Nat Methods

. Author manuscript; available in PMC 2009 October 1.

NIH-PA Author Manuscript

and E. Isacoff, a NSF NIH grant to P. Perona and M.J. Dickinson, and a postdoctoral fellowship of the Alexander von

Humboldt-Foundation to H. Dankert. D.J. Anderson is an Investigator of the Howard Hughes Medical Institute. We

thank M. Heisenberg for German sponsorship of H. Dankert, and for sharing information and data regarding aggression

arenas and automated assays.

References

1. Manoli DS, Foss M, Villella A, Taylor BJ, Hall JC, Baker BS. Male-specific fruitless specifies the

neural substrates of Drosophila courtship behaviour. Nature 2005;436:395–400. [PubMed: 15959468]

2. Demir E, Dickson BJ. fruitless splicing specifies male courtship behavior in Drosophila. Cell

2005;121:785–794. [PubMed: 15935764]

3. Stockinger P, Kvitsiani D, Rotkopf S, Tirian L, Dickson BJ. Neural circuitry that governs Drosophila

male courtship behavior. Cell 2005;121:795–807. [PubMed: 15935765]

4. Vrontou E, Nilsen SP, Demir E, Kravitz EA, Dickson BJ. fruitless regulates aggression and dominance

Drosophila

. Nat Neurosci 2006;9:1469–1471. [PubMed: 17115036]

5. Manoli DS, Meissner GW, Baker BS. Blueprints for behavior: genetic specification of neural circuitry

for innate behaviors. Trends Neurosci 2006;29:444–451. [PubMed: 16806511]

6. Callaway EM. A molecular and genetic arsenal for systems neuroscience. Trends Neurosci

2005;28:196–201. [PubMed: 15808354]

7. Luo L, Callaway EM, Svoboda K. Genetic dissection of neural circuits. Neuron 2008;57:634–660.

[PubMed: 18341986]

8. Suh GS, Wong AM, Hergarden AC, Wang JW, Simon AF, Benzer S, Axel R, Anderson DJ. A single

population of olfactory sensory neurons mediates an innate avoidance behaviour in Drosophila. Nature

2004;431:854–859. [PubMed: 15372051]

9. Katsov AY, Clandinin TR. Motion processing streams in Drosophila are behaviorally specialized.

Neuron 2008;59:322–335. [PubMed: 18667159]

10. de Bono M, Maricq AV. Neuronal substrates of complex behaviors in C. elegans. Annual review of

neuroscience 2005;28:451–501.

11. Skrzipek KH, Kroner B, Hager H. Inter-Male Aggression in Drosophila-Melanogaster - Laboratory

Study. J Comp Ethol 1979;49:87–103.

12. Hoffmann AA. A laboratory study of male territoriality in the sibling species Drosophila melanogaster

and D. simulans Anim Behav 1987;35:807–818.

13. Chen S, Lee AY, Bowens NM, Huber R, Kravitz EA. Fighting fruit flies: a model system for the

study of aggression. Proc Natl Acad Sci USA 2002;99:5664–5668. [PubMed: 11960020]

14. Kravitz EA, Huber R. Aggression in invertebrates. Curr Opin Neurobiol 2003;13:736–743. [PubMed:

14662376]

15. Greenspan RJ, Ferveur JF. Courtship in Drosophila. Annu Rev Genet 2000;34:205–232. [PubMed:

11092827]

16. Wehrhahn C, Poggio T, Bülthoff H. Tracking and Chasing in Houseflies (

Musca

). Biol Cybern

1982;45:123–130.

17. Branson K, Belongie S. Tracking multiple mouse contours (without too many samples). IEEE

Computer Vision and Pattern Recognition 2005;1:1039–1046.

18. Khan Z, Balch T, Dellaert F. MCMC-Based Particle Filtering for Tracking a Variable Number of

Interacting Targets. IEEE Trans Pattern Analysis Machine Intelligence 2005;27:1805–1819.

19. Veeraraghavan A, Chellappa R, Srinivasan M. Shape-and-Behavior Encoded Tracking of Bee Dances.

IEEE Trans Pattern Analysis Machine Intelligence 2008;30:463–476.

20. Fry SN, Rohrseitz N, Straw AD, Dickinson MH. TrackFly: Virtual reality for a behavioral system

analysis in free-flying fruit flies. J Neuroscience Methods 2008;171:110–117.

21. Wolf FW, Rodan AR, Tsai LT, Heberlein U. High-resolution analysis of ethanol-induced locomotor

stimulation in Drosophila. J Neurosci 2002;22:11035–11044. [PubMed: 12486199]

22. Valente D, Golani I, Mitra PP. Analysis of the Trajectory of

Drosophila melanogaster

in a Circular

Open Field Arena. PLoS ONE 2007;2(10):e1083.10.1371/journal.pone.0001083 [PubMed:

17957265]

Dankert et al.

Page 8

Nat Methods

. Author manuscript; available in PMC 2009 October 1.

NIH-PA Author Manuscript

23. Hoyer SC, Eckart A, Herrel A, Zars T, Fischer SA, Hardie SL, Heisenberg M. Octopamine in male

aggression of Drosophila. Curr Biol 2008;18:159–167. [PubMed: 18249112]

24. Wang L, Dankert H, Perona P, Anderson DJ. A common genetic target for environmental and heritable

influences on aggressiveness in Drosophila. Proc Natl Acad Sci USA 2008;105:5657–5663.

[PubMed: 18408154]

25. Dierick HA. A method for quantifying aggression in male Drosophila melanogaster. Nat protocols

2007;2:2712–2718.

26. Bishop, CM. Pattern Recognition and Machine Learning. Springer; New York: 2007. p. 738

27. Otsu N. A threshold selection method from gray level histograms. IEEE Trans Systems, Man and

Cybernetics 1979;9:62–66.

28. Johns DC, Marx R, Mains RE, O’Rourke B, Marban E. Inducible genetic suppression of neuronal

excitability. J Neurosci 1999;19:1691–1697. [PubMed: 10024355]

29. Ferveur JF, Stortkuhl KF, Stocker RF, Greenspan RJ. Genetic feminization of brain structures and

changed sexual orientation in male Drosophila. Science 1995;267:902–905. [PubMed: 7846534]

30. Certel S, Savella MG, Schlegel DCF, Kravitz EA. Modulation of Drosophila male behavioral choice.

Proc Natl Acad Sci USA 2007;104:4706–4711. [PubMed: 17360588]

Dankert et al.

Page 9

Nat Methods

. Author manuscript; available in PMC 2009 October 1.

NIH-PA Author Manuscript

Figure 1. Imaging setup for genetic screens in

Drosophila

. (a)

Lateral cut through our double-chamber (all lengths in millimeter).

(b)

Example of a high-

throughput behavioral screening assay - 4 double-chambers, 4 cameras, 2 PCs, and standard

video-acquisition software.

(c)

Double-chamber with the walls removed to expose the floor.

(d)

Camera view of the double chamber. Each of the two arenas has food in the centre,

surrounded by agarose; walls are coated with Fluon. Bar, 10 mm.

Dankert et al.

Page 10

Nat Methods

. Author manuscript; available in PMC 2009 October 1.

NIH-PA Author Manuscript

Figure 2. Detection and tracking of fruit flies. (a–c)

Fly segmentation procedure.

(a)

‘Foreground image’

, computed by dividing the original

image

by (

+ 3

) (

values in false-colors).

(b)

The fly body is localized by fitting a

Gaussian mixture model

(GMM) with three Gaussians (black curves; background (dashed),

other parts and body (solid)) to the histogram of

values (gray curve) using the Expectation

Maximization (EM) algorithm

. First (top) and final (bottom) iterations of the GMM-EM

optimization. All pixels with brightness values greater than the value at the intersection of the

solid black Gaussians (orange areas) are assigned to the body, and are fit with an ellipse.

(c)

Full fly detection by segmenting the complete fly from the background, with body parts and

wings (empirically represented as four segments/colors)

. First iteration (top) and final result

(bottom).

(d)

Head and abdomen are resolved by dividing the fly along the minor axis

and

comparing the brightness-value distribution of both halves (head is brighter; see also

). The

wings (

) are measured by detecting, on each side of the fly’s posterior half, the

Dankert et al.

Page 11

Nat Methods

. Author manuscript; available in PMC 2009 October 1.

NIH-PA Author Manuscript

pixel with the furthest distance from the center of the ellipse (wing tip) in the segmented full

fly.

(e)

Definition of fly orientation

and moving direction

move

(f)

Separation of occluding

(pair 1) and touching flies (pair 2). Original image (left), foreground image (center), and final

segmentation result (right) with the corresponding ellipses.

(g)

Definition of additional

features. Bars, 1 mm.

Dankert et al.

Page 12

Nat Methods

. Author manuscript; available in PMC 2009 October 1.

NIH-PA Author Manuscript

Figure 3. Detectable actions

(

a–d)

Single shots of side and top views of

(a)

lunging,

(b)

tussling,

(c)

wing threat, and

(d)

copulation. High-resolution sequences of

(e)

wing extension and circling, as well as

(f)

chasing

(see also Supplementary Videos 1–

online). Time index in seconds is relative to the first frame

in each movie clip. Bars, 1 mm. Supplementary Figure 1 online shows the same actions as

detected by our system.

Dankert et al.

Page 13

Nat Methods

. Author manuscript; available in PMC 2009 October 1.

NIH-PA Author Manuscript

Figure 4. Performance of action detection. (a)

Performance of our lunge detector, described by the Receiver Operating Characteristic (ROC).

Each ROC curve gives the fraction of false-negatives (number of missed lunges divided by the

total number of lunges on the ground truth) vs the number of false-positives. Curves are shown

for different values of k-nearest neighbors constant

. Best performance is achieved for

= 15.

The operating point of the system is shown by a black circle; it is obtained by labeling an action

‘lunge’ when 12 or more of its

= 15 nearest-neighbors are lunges. The performance of an

expert human observer (40% missed lunges) is indicated by the gray cross. The expert detected

84 lunges, while our two-step process for establishing ground-truth (see

Methods

) yielded 139

lunges.

(b)

Comparison between automatically measured lunges and ground-truth. Each dot

represents the number of lunges detected automatically (

axis) vs the number of lunges in the

ground truth (

axis) for each of 56 20-minute movies of fly pairs. Note that automatic counting

is very close to ground-truth both when there are many lunges and when there are few.

Dankert et al.

Page 14

Nat Methods

. Author manuscript; available in PMC 2009 October 1.

NIH-PA Author Manuscript

Figure 5. Genetic and environmental influences on aggressive and courtship behavior. (a,g)

Number of lunges,

(b,h)

tussling,

(c,i)

lunges per meter,

(d,j)

wing threats,

(e)

circling,

(f)

wing

extensions. All panels show mean and standard error of the mean.

(a–d)

Octopamine control

(tdc2/+) and mutant (tdc2/kir).

(e–j)

CS male-male, male-female, and

Cha-Gal4;UAS-tra

(“Cha-Tra”) male-male pairs.

(k–m)

Ethograms, based on transition matrices. The ethograms

show transitions where the interval between a fly’s action and the next lasted

≤

10 seconds.

We count intervals > 10 seconds without action as ‘no action’ nodes (not shown). The transition

probability is represented by the thickness of the arrows (normalized over all arrows that exit

a node including the arrow into ‘no action’). The arrow stumps represent the transition

probability from one action into the same action. Circle diameters (logarithmically scaled) and

Dankert et al.

Page 15

Nat Methods

. Author manuscript; available in PMC 2009 October 1.

NIH-PA Author Manuscript

numbers denote the average action frequencies.

(n)

Frequency of CS male-fly positions while

extending a wing towards a decapitated CS female. Left: group-housed flies (GG, n = 10);

right: single-housed flies (SS, n = 10).

(o)

Dankert et al.

Page 16

Nat Methods

. Author manuscript; available in PMC 2009 October 1.

NIH-PA Author Manuscript

Dankert et al.

Page 17

Table 1

Performance evaluation of action detection.

# of fly

pairs

# of events

correct positives

false negatives

# of false

positives

# of false positives

# of false

positives

Behavior

in percent

per 20

′

movie

per event

Lunging

a, b

139

90.7

9.3

0.05

Tussling

176

0.33

0.07

Wing Threat

a,c

94.3

5.7

0.1

0.04

Wing Extension

d,e

797

96.7

3.3

3.5

0.04

Circling

d,f

422

99.8

0.2

1.8

0.04

Chasing

400

98.0

2.0

0.67

0.01

wild-type (CS) male-male fly pairs.

56 additional pairs were tested and the correlation with ground-truth was 0.99 (see text).

118 hand-counted wing threats. 87/118 lasted longer than 0.3s. False-positives are ambiguous situations of wing threat or common wing extension.

wild-type (CS) male-female fly pairs.

906 hand-counted single wing extensions. 797/906 lasted longer than 1 s. False-positives were due to segmentation errors.

422/435 hand-counted circling events had a minimum length of 1 s.

Cha-Gal4;UAS-tra

male-male fly pairs. Minimum duration 1 s.

Nat Methods

. Author manuscript; available in PMC 2009 October 1.