of 12
1305
INTRODUCTION
Capturing the detailed locomotory kinematics and behavior of
biological organisms plays an important role in a wide variety of
research disciplines (Fry et al., 2003; Gray, 1933; Park et al., 2001;
Pfau et al., 2005). Many studies in biomechanics, neuroethology
and developmental biology rely on the analysis of video sequences.
Such analyses stand to benefit greatly from computer vision
techniques, which can accurately and efficiently screen complex
morphometric and locomotory characteristics.
Following the pioneering work of Eadweard Muybridge
(Muybridge, 1887), scientists have developed a wide range of
methods to record and quantify animal locomotion. Many studies
have relied on tracking landmarks, such as leg joints (Full and Tu,
1991) or markers on the animal’s body (Hedrick et al., 2004; Standen
and Lauder, 2005). However, it is not always feasible or advisable
to attach markers, e.g. during field observations or on animals that
have a mucous layer (e.g. many fish) or motile skin (e.g. horses).
In these cases, tracking the silhouette of the animal or a body part
can offer an alternative [e.g. silhouettes of ascidian larvae and
swimming fish (McHenry, 2001; Tytell and Lauder, 2002)].
Tracking swimming movements by filming the swimmer’s
silhouette has a long tradition (Batty, 1984; Gray, 1933). Silhouettes
lend themselves to automatic tracking (Tytell and Lauder, 2002).
Furthermore, tracking routines can be refined by using information
about the shape of the animal or limb to be tracked [e.g. insect wings
(Fry et al., 2003; Willmott and Ellington, 1997)]. Error reduction
as a result of including shape information is particularly valuable
when studying the propulsive body wave of fishes. The exact shape
of a fish’s body wave depends on the interaction between the solid
body of the fish and the surrounding water. Exact knowledge of the
wave shape can provide valuable insight into swimming performance
and the interaction between undulating body and surrounding water
(Cheng and Pedley, 1998; McHenry et al., 1995; Wainwright, 1983).
For example, the stiffness of the fish’s body affects the shape of
the body wave and thereby influences the propulsive performance
of the fish. An interesting case study is presented by the
stocksteif
mutation in zebrafish, which is characterized by an overossification
of the notochord. The axial skeleton of these mutants is a stiff, bony
rod, contrasting the flexible series of articulating vertebrae in their
wild-type siblings.
The zebrafish has long served as a convenient model to study the
various aspects of fish swimming (Fuiman and Webb, 1988; Thorsen
et al., 2004). Automated tracking and analysis systems have been
previously developed for zebrafish (Bang et al., 2002; Blaser and
Gerlai, 2006). However, these systems track the fish only as a point
and cannot quantify body wave kinematics of swimming. Other
studies of zebrafish swimming have manually tracked the fish to
quantify the body posture, a method that is both time-consuming
and potentially prone to subjective errors (Budick and O’Malley,
2000; McElligott and O’Malley, 2005; Müller and van Leeuwen,
2004). Tytell and Lauder used a semi-automated method to estimate
the fish midline by manually indentifying the snout and tail and
The Journal of Experimental Biology 211, 1305-1316
Published by The Company of Biologists 2008
doi:10.1242/jeb.010272
Automated visual tracking for studying the ontogeny of zebrafish swimming
Ebraheem Fontaine
1,
*, David Lentink
2
, Sander Kranenbarg
2
, Ulrike K. Müller
3
, Johan L. van Leeuwen
2
,
Alan H. Barr
4
and Joel W. Burdick
1
1
Mechanical Engineering, California Institute of Technology, Pasadena, CA 91125, USA,
2
Experimental Zoology Group,
Wageningen University, Wageningen, The Netherlands,
3
Department of Biology, California State University Fresno, Fresno, CA
93740, USA and
4
Computer Science, California Institute of Technology, Pasadena, CA 91125, USA
*Author for correspondence (e-mail: ebraheem@robotics.caltech.edu)
Accepted 21 February 2008
SUMMARY
The zebrafish
Danio rerio
is a widely used model organism in studies of genetics, developmental biology, and recently,
biomechanics. In order to quantify changes in swimming during all stages of development, we have developed a visual tracking
system that estimates the posture of fish. Our current approach assumes planar motion of the fish, given image sequences taken
from a top view. An accurate geometric fish model is automatically designed and fit to the images at each time frame. Our
approach works across a range of fish shapes and sizes and is therefore well suited for studying the ontogeny of fish swimming,
while also being robust to common environmental occlusions. Our current analysis focuses on measuring the influence of
vertebra development on the swimming capabilities of zebrafish. We examine wild-type zebrafish and mutants with stiff vertebrae
(
stocksteif
) and quantify their body kinematics as a function of their development from larvae to adult (mutants made available by
the Hubrecht laboratory, The Netherlands). By tracking the fish, we are able to measure the curvature and net acceleration alon
g
the body that result from the fishʼs body wave. Here, we demonstrate the capabilities of the tracking system for the escape
response of wild-type zebrafish and
stocksteif
mutant zebrafish. The response was filmed with a digital high-speed camera at
1500
·
frames
·
s
–1
. Our approach enables biomechanists and ethologists to process much larger datasets than possible at present.
Our automated tracking scheme can therefore accelerate insight in the swimming behavior of many species of (developing) fish.
Supplementary material available online at http://jeb.biologists.org/cgi/content/full/211/8/1305/DC1
Key words: tracking, geometric modeling, estimation, zebrafish, swimming.
THE JOURNAL OF EXPERIMENTAL BIOLOGY
1306
automatically estimating the midline from the extracted silhouette
(Tytell and Lauder, 2002). Other authors have relied on
‘skeletonizing’ algorithms that dissolve a binary image representing
the animal’s silhouette down to its midline (Cronin et al., 2005; Geng
et al., 2004; McHenry, 2001). Although this approach is automated,
it will not estimate the correct midline if other objects are present
in the binary image because it cannot distinguish between pixels that
belong to the animal’s silhouette and those that belong to a different
object. As a result, these algorithms will not correctly estimate the
fish’s body posture when there is environmental clutter such as other
fish, plants, or a hair used to initiate behavioral responses, as we did
in our recordings. Automated kinematic analysis of multiple zebrafish
larvae was recently demonstrated. However, this particluar analysis
technique utilizes an image filter that is customized for the appearance
of zebrafish larvae of a specific age (Burgess and Granato, 2007).
This technique does not extend nicely to zebrafish of different ages,
other fish species, or when environmental clutter is present.
Here, we present a complete method for accurately and efficiently
quantifying the body posture of zebrafish and other organisms with
symmetric medial profiles. Our approach directly models the shape
of the animal and utilizes locations of high contrast in the image to
estimate its posture. The posture estimate is calculated using
techniques that remain robust to clutter. The detailed swimming
motion is estimated based on dorsal images of the fish recorded at
sufficiently high frame rate (Harper and Blake, 1989), which
enables a quantitative evaluation of the animal’s kinematics and
dynamics (provided the mass distribution of the animal is known).
In the first section, we review the overall approach for performing
model-based visual tracking. Subsequently, we develop a detailed
geometric model for the zebrafish, including appropriate motion and
measurement models that use information from the previous and
current frame to estimate the fish’s current position and posture.
Finally, we demonstrate the capabilities of our tracking approach
on zebrafish performing an escape response at three stages during
their development (from larvae to juvenile).
MATERIALS AND METHODS
Zebrafish (
Danio rerio
Hamilton 1822) eggs were collected after
mating one
stocksteif
heterozygous female with one
stocksteif
heterozygous male. The batch of eggs contained both
stocksteif
mutant and wild-type embryos, but the mutant phenotype does not
become apparent until 5 days post fertilization (d.p.f.). The embryos
were reared at the optimal rearing temperature of 28°C. After
hatching, the embryos were fed
Paramecium
(5 and 6
·
d.p.f.) and
Artemia
(from day 7 onwards). A fast startle response was recorded
at 5, 15 and 28
·
d.p.f. for both wild-type and
stocksteif
mutant
animals
using a high-speed video camera (Photron, APX RS,
1500
·
frames
·
s
–1
, 1024

1024
·
pixels, exposure time 1/8000
·
s) fitted
with a 105
·
mm Nikon lens. The startle responses were elicited by
touching the animals with a horse hair. We analyzed recorded
sequences from the initiation of the escape response to the moment
when the fish either leaves the field of view or ceases active
swimming. Our sequences therefore include stages 1 and 2 of an
escape response, and usually several tail beats that are part of stage
3 (Weihs, 1973) (Movie 1 in supplementary material).
Model-based image tracking and nonlinear estimation
Our goal was to quantify the body wave and swimming kinematics
of a zebrafish from video sequences recorded from a top view. First,
we built a geometric model of the zebrafish that is defined by the
vector
p
, which contains the parameters that encode the body wave
shape and location of the model. Then, tracking is performed by
recursively estimating the fish parameters
p
k
from measurements in
the image,
z
k
at time step
k
. This tracking approach is formulated
within a discrete time (
k
), dynamic state space, framework as:
p
k
=
f
(
p
k
–1
,
ξ
k
–1
)
·
,
z
k
=
h
(
p
k
,
ν
k
)
·
,(
1)
where
f
is the motion model,
h
is the measurement model and
ζ
k
and
ν
k
are independent and identically distributed noise processes (models
of the noise present in our image sequence). From a Bayesian
perspective, this corresponds to estimating the probability density
function
P
of the fish parameters given all of the image measurements
up to the current time step (i.e. the posterior)
via
Bayes’ rule:
P
(
p
k
|
z
1:
k
) = [
P
(
z
k
|
p
k
)
P
(
p
k
|
z
1:
k
–1
)] /
P
(
z
k
|
z
1:
k
–1
)
·
.(2)
The optimal estimate of the fish’s shape and position will then
correspond to the conditional mean of this probability density
function,
p
k
=E[
p
k
|
zk
]=

p
k
P
(
p
k
|
z
1:
k
)d
p
k
. In general, this recursive
solution to the tracking problem is intractable and approximate
solutions must be used instead. These approximation methods can
primarily be broken into two categories: those that assume the
probability density functions are normal distributions [e.g. Kalman
Filters (van der Merwe and Wan, 2003)] and those that allow the
probability density functions to assume arbitrary distributions [e.g.
particle filters (Doucet et al., 2001)]. Although particle filters are
able to solve very general estimation problems, there are many
problems within visual tracking when the normal assumption holds,
in which case Kalman filters provide accurate solutions and
computational efficiency. The present study demonstrates that the
normal assumption holds for estimating the location of zebrafish
from top view images within a constrained laboratory that contains
few environmental occlusions, hence we adopt Kalman filtering.
Recently, the Kalman filter has been considerably improved by
statistical linearization and now performs well when applied to
nonlinear motion and measurement models (Ito and Xiong, 2000;
van der Merwe and Wan, 2003; Nörgaard et al., 2000; Sibley et al.,
2006). These Sigma Point Kalman Filters (SPKF) recursively
estimate the optimal fish parameters and their uncertainty by taking
the weighted average of a set of regression points drawn from the
probability density function that describes the fish’s shape and
location at the previous time step. In this study, we leverage the power
of SPKFs to solve our nonlinear estimation problem. However, in
order to estimate the fish parameters, we must also develop equations
for the motion model
f
and the measurement model
h
that make sense
for a swimming zebrafish during development. In the following
sections we apply a two-step approach. First we construct a flexible
geometric model of a zebrafish outline. Second, we develop
appropriate motion and measurement equations for this model, which
are used in conjunction with the SPKF to automatically optimize the
fish parameters based on the processed image data.
Geometric modeling
The motion of zebrafish in the shallow water of a tank is largely
planar, so we model the fish motion as restricted to two dimensions
and assume orthographic projection in the camera model. We also
assume that the zebrafish maintains a constant length and width profile
during swimming. Therefore, we model the centerline of the fish as
a planar, inextensible curve, which can only undergo bending. The
body shape can therefore be described effectively by the bend angle
Θ
of the fish as a function of distance along its length,
u
. This function,
Θ
(
u
), is finitely parameterized using a linear combination of known
basis functions where
Φ
k
j
(
u
), is the
j
th basis function of order
k
. The
current implementation uses eight (
N
Θ
=8) periodic cubic B-splines
as the basis functions, which we found to be sufficient to capture the
E. Fontaine and others
THE JOURNAL OF EXPERIMENTAL BIOLOGY
1307
Automated zebrafish tracking
different bending modes of the swimming zebrafish and mating
Caenorhabditis elegans
(Fontaine et al., 2006). Other choices in the
order and number of basis functions are possible. However,
determining the minimum set of basis functions needed to describe
all possible fish configurations is beyond the scope of this study. To
increase the tracker’s robustness and accuracy, our model assumes
that fish have a stiff head. This assumption prevented the tracker from
creating unrealistic bending deformations in the head region. This
simplification is based on our experimental results (Müller and van
Leeuwen, 2004), which show that the head region of freely swimming
zebrafish undergoes negligible bending (zero local curvature). This
corresponds to
Θ
(
u
) remaining constant in the head region, which we
define as starting at the snout and extending to 20% of the fish’s body
length (length from snout to tail tip or caudal penduncle in the case
of juvenile fish, see Fig.
·
5); in other words, the relative length of the
stiff head region compared with the total length of the fish
L
is
γ
=0.2.
We kept
γ
fixed for our experiments and found that
γ
=0.2 provided
good tracking results in all three considered age groups. We
implemented the head region mathematically by defining the origin
of
u
(where
u
is zero) at a distance of 0.2
·
L
behind the snout, such
that the head region is described by positive
u
values (0≤
u
<0.2
L
) and
the tail region by negative
u
values (–0.8
L
<
u
≤0). This approach
simplifies the formulation of the basis functions
Φ
k
j
(
u
) and the
corresponding local bend amplitudes
α
j
along the fish’s body:
(3)
.
Θ
(
u
)
=
α
j
Φ
j
k
(
u
)
u
0
j
=
1
N
Θ
α
j
Φ
j
k
(0)
u
>
0
j
=
1
N
Θ
Fig.
·
1B illustrates these concepts, i.e. the definition of
u
as well as
how the B-spline basis functions describe the body wave. The spline
bases have local maxima in the tail region, but become constant in
the head region.
Our complete fish model is a symmetric planar mesh with grid
parameters
u
and
v
, which define a local coordinate frame in the
fish body in the length and width directions, respectively. The points
in this mesh [
x
(
u
,
v
)
y
(
u
,
v
)]
T
are continuous functions of
u
and
v
, so
we can evaluate them at whatever discrete grid we choose. For our
experiments, we sample 30 uniformly spaced points in the domain
[–(1–
γ
)
L
,
γ
L
] for
u
and three uniformly spaced points in the domain
[–1, 1] for
v
. Because the number of sampled points in the (
u
,
v
)
grid remains constant during tracking, the mesh points that make
up our fish model become functions of the position and shape
parameters,
p
. We construct the centerline of our symmetric mesh
by integrating the unit tangent vector
e
1
along the grid parameter
u
, and the width is created by expanding the centerline in the
direction of the normal vector
e
2
according to the value of
R
(
u
),
which is the fish’s width as a function of distance along its length.
R
(
u
) is defined as a fourth-order periodic B-spline function using
20 basis functions, and its value is calculated from the first frame
of the video recording (see ‘Initial detection of zebrafish’) and held
fixed during tracking. This process is illustrated by Fig.
·
2 and further
details are provided in the Appendix. We denote the complete fish
model as
H
(
p
)
where
p
[
α

T
]
T
are the fish parameters, which include
the bend angle amplitudes
α

defined earlier and
T
, the global
translation vector of the entire fish.
Creating deformable models based on medial profiles has been
used in segmentation problems in medical imaging (Hamarneh and
McInerney, 2001) and for tracking multiple
C. elegans
from
0
0.2
0.4
0.6
0.8
Φ
k
j
(
u
)
No
bending
L
γ
L
u
0
y
x
2
(
u
)
Θ
(
u
)
T
H
(
p
)
A
B
u
ε
[–(1–
γ
)
L
,
γ
L
]
γ
ε
[0,1]
1
(
u
)
e
e
Fig.
·
1. (A,B) Illustration of the modeling approach
used for zebrafish. (A) Geometric mesh
H
(
p
) (green)
with local tangent (
e
1
) and normal (
e
2
) vectors used
to construct the mesh. The parameter
T
and the
function
Θ
(
u
) define the position and shape of the
model and are estimated during tracking. (B) Head
region of length
γ
L
is designated as rigid, while tail
region bends according to linear combination of eight
B-spline bases
Φ
j
k
(
u
).
=

d
u
−2
−1
0
1
+
Centerline
−1.5
−1
−0.5
0
0.5
1
x
y
−4
−3
−2
−1
0
1
−0.5
0
0.5
u
±
R
(
u
)
−1.5
−1
−0.5
0
0.5
1
1
[
Θ
(
u
)]
Θ
(
u
)
−4 −3 −2 −1 0
1
u
α
j
−3
−2
−1
0
−2.5
−1.5
−0.5
x
y
−3
−2
−1
0
−2.5
−1.5
−0.5
0.5
H
(
p
)
e
Fig.
·
2. Method for constructing fish model used in this analysis. The tangent vector associated with the function
Θ
(
u
) is integrated to create the fish
centerline. This centerline is combined with the fishʼs width profile
R
(
u
) to create the complete model
H
(
p
).
R
(
u
) remains fixed during tracking, and the
bending of the model is modulated by changing the values of
α
j
, which control the shape of
Θ
(
u
).
THE JOURNAL OF EXPERIMENTAL BIOLOGY
1308
microscopy images (Fontaine et al., 2006; Roussel et al., 2007).
Because zebrafish are laterally symmetric about their body axis, our
medial profile representation offers several advantages for the
tracking framework. Each B-spline basis function is defined only
over a subregion of the fish body. Therefore, the local bend
amplitudes,
α
j
, have local control over the degree of bending in the
body. This property is analogous to the fish anatomy, where
contractions of individual muscles affect the bending over subregions
of the fish’s body. In summary, our parameterization of the
centerline has few degrees of freedom, requires no training data,
and offers a natural and anatomically sound way to constrain the
fish’s length and designate certain regions as stiff.
Motion model
The motion model of the fish predicts the fish parameters at the
current time step based on the parameters calculated at the previous
time step. By predicting the motion, it provides a better initial
estimate of the fish parameters before they are updated using the
measurement model. We assume that the fish movement is a
combination of undulatory motion along its body and a displacement
of the whole body along the centerline axis. The undulatory motion
of steady swimming in fish consists of a traveling wave of increasing
amplitude from head to tail. Given our model of the zebrafish, this
motion is governed by the change in local bend amplitudes,
α
j
, from
the previous time step to the current one. Given that our video
sequences are filmed at a sufficiently high frame rate, we make the
assumption that the current local bend amplitudes are equal to the
previous ones plus a random variable drawn from a zero mean
normal distribution. This is a simple way to allow in the model that
we are more uncertain about the body wave shape of the fish since
it has moved between frames. We further assume that the fish has
constant velocity,
η
1
, between frames during displacement along
its body axis that is corrupted by acceleration noise,
η
2
. This velocity
is also estimated and thus incorporated into the set of fish parameters
p
=[
α

T
η
1
]
T
. The complete motion model calculates the current fish
parameters after the fish has undergone a total axial displacement
of
η
0
=
η
1
Δ
t
+
η
2
(
Δ
t
2
/2) from the previous frame, where
Δ
t
is the time
step between subsequent frames (inverse of the camera frame rate)
and
η
2
is modeled as a zero mean random variable drawn from a
normal distribution with fixed variance. An overview of the motion
model is shown in Fig.
·
3 with displacements exaggerated for
illustration purposes.
E. Fontaine and others
η
0
Axial displacement
p
k
–1
Axial displacement
+
u
nd
u
latory motion
p
k
Fig.
·
3. Illustration of motion model of the fish. We assume that the total
motion between frames
k
–l and
k
can be decomposed into undulatory
motion and axial displacement. Note that figure displacements are
exaggerated for illustration purposes. Actual motion between frames is
much smaller due to the high frame rate of the camera.
-
Detected edge points
AB
Fig.
·
4. Measurement model for matching zebrafish images. (A) Initial
estimate of the model location (white broken line) with matching edge
feature points,
r
i
(black filled, white circles). Red lines denote the 1D search
regions for edge points. Note the tail is initially not matched to the
boundary. (B) Final estimate of the model after four iterations. Although
some error is present between the outline of the model and the actual fish,
the centerline is accurately estimated based on visual inspection. Errors in
the outline are due to small out of plane motions of the fish.
No edge
meas
u
rements
Fig.
·
5. At age 28 days, the fish has fully developed pectoral and caudal
fins, which can cause incorrect model fitting if they are mistakenly classified
as part of the boundary. To address this, we modify the juvenile fish model
so that it does not take edge measurements in the pectoral and caudal
regions.
A
−0.5
0
0.5
−0.4
−0.2
0
0.2
0.4
0.6
0.8
u
(mm)
Width profile (mm)
Original,
B
R
Original,
B
L
Modeled,
R
(
u
)
B
Fig.
·
6. (A,B) Illustration of the initialization process used in the fish tracker.
(A) The initial fish centerline (white),
C
(
u
), is estimated from the left (blue)
and right (red) fish outlines. (B) This is used to estimate the width profile
R
(
u
) from the raw pixel data,
B
R
and
B
L
. Our modeling approach assumes
a symmetric fish. Figure is zoomed into the head region because
R
(
u
) and
pixel data are indistinguishable in the tail region.
THE JOURNAL OF EXPERIMENTAL BIOLOGY
1309
Automated zebrafish tracking
Measurement model
The present tracker system assumes that we are able to measure
the location of the fish boundary, so our measurement model
consists of the boundary points,
q
i
=[
x
i
y
i
]
T
, along with their
outward normal vectors,
n
i
=[
n
i
x
n
i
y
]
T
along the fish’s outline. In
order to fit our geometric model to the image at each frame, we
must take the appropriate image measurements and match them
with corresponding locations in the model. To achieve this, we
first segment the image using background subtraction, which
produces a binary image that is used to search for edges. Next,
we apply a one-dimensional edge-detector filter in the direction
normal to the boundary at each of the boundary points in the fish
model (Blake and Isard, 1998). The distance between
q
i
and the
corresponding detected edge point,
r
i
, is projected onto
n
i
so that
the error minimized by the SPKF is the normal displacement
between the edge points and model points. This process is
illustrated in Fig.
·
4 where the initial estimate, edge points, and
final solution are overlaid on an actual image. By using the SPKF,
we minimize this error to obtain an updated estimate of our fish
parameters
p
k
(see Appendix for details). At age 28 days, the
zebrafish has fully developed pectoral and caudal fins. These fins
can cause incorrect tracking because the lighting conditions can
make them appear as solid as the fish’s body. However, by
modifying the fish model to not take edge measurements in the
pectoral and caudal regions, we are able to accurately estimate the
body posture of the juvenile fish (Fig.
·
5).
Initial detection of zebrafish
Any tracking algorithm relies on an initial estimate of the object
location. To achieve this, we have developed a semi-automated
initialization routine that operates on the first movie frame and
extracts three important pieces of information for tracking: (1) an
estimate of the background image, (2) the initial fish parameters
p
0
,
and (3) the width function
R
(
u
). The background model is estimated
by selecting a region around the fish and erasing it using the built-
in Matlab
TM
function ‘
roifill
’, which smoothly interpolates inward
1 mm
D
1 mm
F
1 mm
A
1 mm
B
1 mm
C
1 mm
E
Fig.
·
7. (A–F) Tracking results
for zebrafish at (A,D) 5
·
d.p.f.,
(B,E) 15
·
d.p.f. and (C,F)
28
·
d.p.f. (see Movie 1 in
supplementary material). The
first row are wild type and the
second row are
s
tock
s
teif
mutants. The raw centerlines
estimated by the tracker are
plotted at 1.3
·
ms intervals for
5 and 15
·
d.p.f. and 2.7
·
ms
intervals for 28
·
d.p.f. Magenta
and yellow trajectories
indicate the paths of the tail
and snout, respectively. Note
in (C,F) that the caudal fin is
not modeled in our current
approach, so its motion is
disregarded.
THE JOURNAL OF EXPERIMENTAL BIOLOGY
1310
from the pixel values on the boundary of the user-defined region.
Next, the background image is used to segment the first movie frame,
and the Matlab
TM
function ‘
bwboundaries
’ calculates the fish
boundary from the resulting binary image. The user is then requested
to click on the snout and tail locations of the fish, which allows us
to divide the boundary into the left and right discrete boundaries of
the fish denoted
B
L
(
i
) and
B
R
(
i
), respectively.
The initial centerline of the fish will be the curve that is
equidistant from the left and right boundaries. To determine this
centerline, we use a modification of the integral area distance used
in (Roussel et al., 2007) which iterates and converges to the fish
centerline. Our method works by first finding the closest point in
B
L
to each point in
B
R
and
vice versa
. Then, we calculate the median
locations between each set of corresponding points in
B
L
and
B
R
.
These median locations are assigned as the new left and right
boundaries and this process is repeated until the boundaries converge
onto the true discrete centerline
C
(
i
) (see Appendix for more details).
The estimated centerline curve
C
(
u
) is determined by fitting a B-
spline through the points calculated from
C
(
i
), and is illustrated in
Fig.
·
6A along with the original left and right boundaries it was
derived from. The bend angle function
Θ
(
u
) is calculated from
C
(
u
),
and initial bending amplitudes are calculated by projecting the bend
angle function onto the basis functions from Eqn
·
3 and illustrated
in Fig.
·
1B. Once the centerline of the fish has been calculated, we
can estimate its optimal radius function
R
(
u
) by minimizing the
normal displacement between the extracted image boundary
B
L,R
and the model boundary dictated by the radius function in the manner
described in Fig.
·
2. The result of this minimization is shown in
Fig.
·
6B and demonstrates that we are able to reconstruct accurately
the width profile of the fish.
RESULTS
In this section we present automated tracking results for wild-type
and
stocksteif
mutant zebrafish at 5, 15 and 28
·
d.p.f. Fig.
·
7 shows
the raw centerlines of the zebrafish estimated by the tracker at fixed
time intervals and demonstrates the quality of the proposed method.
Our method successfully tracks fast escape responses of fish larvae
(Fig.
·
7A,D) despite occasional partial occlusions [in this case, the
hair used to induce the escape response (Fig.
·
7E)]. Furthermore,
tracking takes an average computation time of 5.5±1.7
·
s
·
frame
–1
on
a 3.0
·
GHz Intel
®
Xeon processor, which enables us to analyze large
datasets very quickly. To calculate an upper bound on the accuracy
of our tracking approach, we created a synthetic movie sequence
by rendering our model along a known trajectory. We then tracked
this synthetic sequence and measured the average error between the
known and estimated centerlines over time (Fig.
·
8). We were able
to localize the synthetic data within 0.5% of the body length on
average; our errors in estimating the real movie sequences will be
slightly larger because they contain additional noise sources not
present in the synthetic one.
From these centerlines, we wish to extract important kinematic
parameters to gain insight into developmental influences on the
propulsion mechanisms of swimming fish, and
vice versa
: the
mechanical influences on the development of the fish. One of these
parameters is body axis curvature, which provides information about
the muscle strains the fish undergoes. To measure curvature, we apply
spatial smoothing to the extracted centerlines and then apply temporal
smoothing directly to the curvature values. Spatial smoothing is
E. Fontaine and others
0
0.2
0.4
0.6
0.8
1
0
0.002
0.004
0.006
0.008
0.01
Position along the body axis
Centerline RMS error
A
B
Fig.
·
8. Error estimates from tracking synthetic images generated with our
model (A). This provides an upper bound on the accuracy that we can
achieve with the current implementation. (B) Given noiseless images, we
can localize the centerline of the fish to within 0.5% of its body length on
average. Actual errors on real data will be slightly larger than this.
0
0.5
1
1.5
2
2.5
5 d.p.f.
15 d.p.f.
28 d.p.f.
Wild type
0
0.2
0.4
0.6
0.8
1
0
0.05
0.1
0.15
0.2
Normalized position along body axis
C
u
rvat
u
re RMS
error (

10
–3
)
Centerline RMS
error (

10
–3
)
Stocksteif
A
B
C
D
0
0.2
0.4
0.6
0.8
1
Fig.
·
9. Average error between the filtered and
unfiltered data as a function of body axis
position. The error for the centerline location
(A,C) and curvature (B,D) are normalized to
body length and measured at 51 uniformly
spaced locations along the fish body. This
provides an average deviation over time
between the filtered and unfiltered data at
particular locations along the fish. Small
values are achieved for both wild-type and
s
tock
s
teif
fish, illustrating that our post-
processing filtering technique retains most of
the original information.
THE JOURNAL OF EXPERIMENTAL BIOLOGY