1
Accurate
cell tracking and lineage construction
in live
-
cell imaging experiments
with deep learning
Erick Moen
1
, Enrico Borba
2
, Geneva Miller
1
,
Morgan Schwartz
1
,
D
ylan Bannon
1
,
N
ora Koe
3
, Isabella Camplisson
2
,
Daniel Kyme
2
, Cole Pavelchek
4
, T
yler
Price
1
,
T
akamasa Kudo
5
, Edward Pao
1
,
William Graf
1
, David Van Valen
1,
†
1. Division of Biology and Bioengineering, California Institute of Technology
2.
Department of Computer Science, California Institute of Technology
3. Department of
Electrical Engineering, Cal
ifornia Institute of Technology
4. Department of
Neuroscience
s
,
University of California, San Diego
5
. Department of Chemical and Systems Biology, Stanford University
†: Corresponding author
–
vanvalen@caltech.
edu
Abstract
Live
-
cell imaging experiments have opened an exciting window into the behavior of living systems. While these
experiments can produce rich data, the computational analysis of these datasets is challenging. Single
-
c
ell analysis
requires that
cells be accurately identified in each image and subsequently tracked over time. Increasingly, deep
learning is being used to interpret microscopy image with single cell resolution. In this work, we apply deep learning
to the pro
blem of tracking single cel
ls in live
-
cell imaging data. Using crowdsourcing and a human
-
in
-
the
-
loop
approach
to
data annotation, we constructed a dataset of
over
1
1
,000
trajectories
of cell nuclei
that includes lineage
information. Using this dataset, we successfully trained a deep
learning model to perform cell tracking within a linear
programming framework. Benchmarking tests demonstrate that our method achieves state
-
of
-
the
-
art perf
ormance
on the task of cell tracking with respect to multiple accuracy metrics.
Further, we show th
at our deep learning
-
based
method generalizes to perform cell tracking for both fluorescent and brightfield images of the cell cytoplasm, despite
having neve
r been trained
on
those data types.
This
enables analysis of live
-
cell imaging data collected acros
s
imaging modalities.
A persistent cloud deployment of our cell tracker is available at
http://www.deepcell.org
.
Intr
oduction
Live
-
cell imaging experiments, where living cells are imaged over time with fluorescenc
e or brightfield microscopy,
has provided crucial insights into the inner workings of biological systems. To date, these experiments have shed
light on numerous
problems, including information processing in signaling networks
1
–
3
and
quantifying stochastic
gene expression
4
–
7
. One k
ey strength of
live
-
cell imaging
experiments is the ability to obtain dynamic data with
single
-
c
ell resolution. It is now well appreciated that individual cells can vary considerably in their behavior, and the
ability to capture the temporal evolution of c
ell
-
to
-
cell differences has proven essential to understanding cellular
heterogeneity.
Increasing
ly, these dynamic data are being integrated with end
-
point genomic assays to uncover even
more insights into cellular behavior
8
–
10
.
Central to the interpretation of these experiments is image analysis. Traditionally, the analysis of these data
occurs
in three phases. First, images are cleaned with steps that include background subtraction and drift correction. Next,
the i
mage is segmented to identify each individual cell in every frame. This segmentation step can capture the whole
cell or cellul
ar compartments like the nucleus. Lastly, all the detections for an individual cell are linked together in
time to form a temporal
ly cohesive record for each cell
; a schematic of this step is shown in Figure 1
(
a
)
. With a
suitable algorithm
and data structu
re
, these records can contain lineage information such as parent
-
child
relationships for each cell. The output of this analysis pi
peline is a record for each cell of which pixels are associated
with it in each frame of the dataset as well as lineage inform
ation. This record can then be used to obtain quantitative
information
–
ranging from metrics of cellular morphology to fluorescen
ce intensity
–
over time.
Advances in imaging technologies
–
both in microscopes
11
and fluorescent reporters
12
–
have significantly reduced
the difficulty of acquiring live
-
cell imaging data while at the same time
increasing the throughput and the number of
systems amenable to this approach. Increasingly, image analysis is a bottleneck for di
scovery as there is a gap
between our ability to collect and analyze data.
T
his gap can be traced to the limited accuracy and
generality of cell
segmentation and tracking algorithms. These limitations lead to a significant curation time, recently estimated
to be
>100 hours for
one
manuscript worth of data
13
. Recent advances in computer vision, specifically deep learning, are
.
CC-BY-NC-ND 4.0 International license
It is made available under a
(which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
The copyright holder for this preprint
.
http://dx.doi.org/10.1101/803205
doi:
bioRxiv preprint first posted online Oct. 13, 2019;
2
closing this gap
14
. For the purposes of this paper, deep learning refers to a set of
machine learning methods capable
of learning effective representations from data in a supervised or unsupervised fashion. Deep learni
ng has shown a
remarkable ability to extract information from images and it is increasingly being recognized that it is a n
atural fit
for the image analysis needs of the life sciences
15,16
. As a result, deep learning is increasingly being applied to
biological imag
ing data
–
applications include using classification to determine cellular phenotypes
17
, enhancing
image r
esolution
18
, and extracting latent information from brightfield microscope images
19,20
. Of interest
to those
who use live
-
cell imaging has been the application of this technology to single
-
cell segmentation. The popular de
ep
learning model architecture U
-
Net targeted cell segmentation as its first use case
21,22
and our group’s prior work has
shown that deep learning can perform single
-
cell segmentation for organisms
spanning the domains of life as well
as in images of tissues
13,23
.
Recent approaches have extended
th
ese
method
s
to 3D datasets
24
.
The improved accuracy
of single
-
cell segmentations
for live
-
cell imaging
is crucially import
ant, as a single segmentation error in a single
frame can
impair
subsequent attempts at cell tracking and render a cell unsuitable fo
r analysis.
While deep learning has been successfully applied to single
-
cell segmentation, a robust deep learning
-
based ce
ll
tracker
for mammalian cells
has been elusive. Integration of deep learning into live
-
cell imaging analysis pipelines
achieve perfo
rmance boosts by combining the improved segmentation accuracy of deep learning with conventional
object tracking algorithms
13,25
–
27
. These algorithms include line
ar programming
28
and the Viterbi algorithm
29
; both
have seen extensive use on live
-
cell imaging data. While useful, thes
e object tracking algorithms have limitations.
Common eve
nts that lead to tracking errors include cell division and cells entering a
nd leaving the field of view.
Furthermore, their use often necessitates tuning numerous parameters to optimize performance f
or specific
datasets, which leads to fragility on unseen
data.
Though there have been attempts at adapting deep learning to track
ce
lls
30
, their performance is significantly limited by a lack of training data, as
fine
-
tuned conventional methods still
achieve superior performance.
Three technical challenges have impeded the creation o
f a
readily available
, deep learning
-
based cell tracker.
First,
as previously mentioned,
the unique features of live
-
cell
imaging da
ta
(i.e. cell divisions)
confound traditional
tracking methods as well as deep learning
-
based object trackers
.
Second, succe
ssful deep learning solutions are data
hungry. While unsu
pervised deep learning can be a useful tool, most applications of deep lear
ning to imaging data
are supervised and require significant amounts of specialized training data.
Aggregating and curating t
raining data
for tracking
is especially difficult because
of the additional temporal dimension
–
objects must be segmented and
track
ed through every frame of a training dataset. Third,
deep learning’s requirement for hardware acceleration
presents a barrie
r for performing large inference tasks.
On premise comput
ing has limited throughput, while cloud
computing
poses
additional software
engineering challenges
.
In this paper, we address each of these challenges to construct an effective deep learning
-
based so
lution to cell
tracking
in two dimensional live
-
cell imag
ing data
. We show how cell tracking can be solved with deep learning
and
li
near programming
.
We then demonstrate how a combination of crowdsourcing and human
-
in
-
the
-
loop data
annotation can be used t
o create a live
-
cell imaging training dataset consisting
of over 1
1
,000 single cell trajectories.
We benchmark the resulting tracker
using
multiple
metrics and show it achieves state
-
of
-
the
-
art performance on
several datasets, including data from the ISBI
cell tracking challenge. Lastly, leveraging our prior
wor
k
with cloud
computing
31
, we show how our cell tracker
can be integrated into the DeepCell 2.0 single
-
cell image analysis
framework to enable segment
ation
and track
ing
live
-
cell
imaging datasets through their web browser.
Tracking single
cells with deep learning
and linear programming
Our
approach to cell tr
acking is motivated by the now classic work of Jaqaman et al
32
and recent
work
applying deep
learning to o
bject tracking
33
. In
these works
, object tracking
is treated
as a linear assignment problem
(Figure
2
a
)
.
In this framework, N
i
objects in frame i must b
e assigned to N
i+1
objects in frame i+1. To solve this assignment
problem,
one
construct
s
a cost function for a possible pai
ring across frames
, which is traditionally based
on each
object’s location and appearance features (brightness, size, etc.)
28
. The gui
d
ing intuition is that objects are unlikely
to move large distances or have
distinct
changes in appearance from frame
-
to
-
frame if
the
frame rate
is sufficiently
high
. The problem is then reduced to the
selection of
one assignment out of the set of all poss
i
ble assignments that
.
CC-BY-NC-ND 4.0 International license
It is made available under a
(which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
The copyright holder for this preprint
.
http://dx.doi.org/10.1101/803205
doi:
bioRxiv preprint first posted online Oct. 13, 2019;
3
minimizes the cost function
, a
task
that
can be accomplished with the Hungarian algorithm
34
. One complicating factor
of biological object tracking is that objects can appear and disappear
–
this often leads to N
i
and N
i+1
being unequal.
This problem can be solved by introducing a “shadow object” for each object in the two frames
32
–
N
i+1
shadow
objects in frame i and N
i
shadow objects in frame i+1. These shadow objects represent an opportunity for objects to
“die” (if an object in frame i is matched with its shadow object in frame i+1) or t
o
be “born” (if an object in frame i+1
is matched with its shadow object in frame i). This framework leads to a cost matrix describing the cost of every
possible assignment that is size N
i
+ N
i+1
x N
i
+ N
i+1
; its structure is shown in Figure
2
a
.
Assuming e
rror
-
free image segmentation and an accommodation for dealing with cell divisions, cell tracking can fit
neatly into this framework. Whole movies are tracked by sequentially tracking every pair of frames
–
this is to be
contrasted with
approaches like
the
Viterbi algorithm that incorporate multiple frames worth of information to
determine assignments. One advantage of this approach is that it can cope with missing objects
–
instead of using
the objects in fr
ame i for comparison, we instead compare all objec
ts that have been successfully tracked up to frame
i. If objects disappear and reappear, the opportunity to correctly track them still exists. Optimization of the linear
assignment approach’s performance o
n real data comes about through cost function engi
neering. By varying key
aspects of the cost function
–
how sensitive it is to the distance between two cells, how much it weights the
importance of cell movement vs cell appearance, etc.
–
it is possible to
tune the approach to have
acceptable
performance
on live
-
cell imaging datasets. However, this approach
has several downsides
–
the accuracy is limited,
the time required for cost function engineering and curation of results is prohibitive, and solutions a
re tailored to
specific datasets which reduces the
ir generality.
Here, we take a supervised deep learning approach to learn an optimal cost function for the linear assignment
framework. Our approach was inspired by previous work applying deep learning to o
bject tracking
33
. Building on
this work, we make ada
ptations to
deal with the unique features of live
-
cell imaging data
(
Figure
1c
)
. To construct
our learned cost function, we consider it as a classification task. Let us suppose we have two cells
–
cell 1 in frame i
and cell 2 in frame i+1. Our goal is to c
onstruct a classifier that would take in informati
on about each cell and produce
a probability that these two cells are either the same, are different, or have a parent
-
child relationship. If such a
classifier worked perfectly, then we could use it in lieu
of our hand engineered cost function, as is shown
in Figure
2
b
. To incorporate temporal information, we can use multiple frames of information for cell 1 as an input to the
classifier. This allows us access to the temporal information beyond just the two
frames we are comparing. For our
work here, we use
7
frames worth of information.
Our classifier for performing this task is a
hybrid recurrent
-
convolutional
deep learning model; its architecture is
shown in Figure
1c
. This deep learning model takes in 4 p
ieces of information about each cell using 4 separ
ate
branches. Conceptually, each branch seeks to summarize
its
input information as a vector. These summary vectors
can then be fed into a fully connected neural network to classify the two cells being compared. The first branch takes
in the appearance, th
at is a cropped and resized image, of each cell and uses a dee
p convolutional neural network to
generate a summary. This network is applied to every frame for cell 1, creating a sequence of summary vectors.
Conversely, cell 2 only has 1 frame of informatio
n and hence only has 1 summary vector. The appearance gives us
access to information on what the two cells look like, but the resizing operation removes notions of size scale. To
mitigate this, we incorporate a second branch that takes in a vector of morph
ological
information for
each cell and
uses a densely connecte
d neural network to create a summary vector of this information.
The morphology
information used includes
the area, perimeter, and eccentricity.
The third branch acquires information about cell
motion over time. For cell 1, we collect a vector of all the c
entroid displacements. For cell 2, we create a single vector
that is the displacement between cell 1 and cell 2’s centroid. This branch gives us a history of cell 1’s motion and
allows us to see
whether a potential positive assignment of cell 2 would be in
consistent from the point of view of cell
motion. The last branch incorporates neighborhoods, which is an image cropped out around the region surrounding
cell 1. We reasoned that because neighbo
rhoods contain information about cell divisions, they could pr
ove useful in
performing lineage assignments. Just as with appearances, a deep convolutional neural network is used to
summarize the neighborhoods as a vector.
W
e extract the neighborhood
around
the area cell
1
is predicted to be
located given cell 1’s vel
ocity
vectors and
use it as the neighborhood for cell 2
.
The result of these 4 branches are
.
CC-BY-NC-ND 4.0 International license
It is made available under a
(which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
The copyright holder for this preprint
.
http://dx.doi.org/10.1101/803205
doi:
bioRxiv preprint first posted online Oct. 13, 2019;
4
sequences of summary vectors for cell 1 and individual summary vectors for cell 2.
Long short
-
term
mem
ory (LSTM)
layers are then applied to each of cell 1’s sequenc
e of summary vectors to merge the temporal information and create
4 individual vectors that summarize each of the 4 branches. The vectors for cell 1 and cell 2 are then concatenated
and fed into
fully connected layers
.
The final layer applies the softmax t
ransform to
produce the final classification
scores
–
p
same
, p
diff
, and p
parent
-
child
. These three scores, which are all positive and sum to 1, can be thought of as
probabilities. They are used
to construct the cost matrix, as shown in Figure 2
b
. If a cell
in frame i+1 is assigned to a
shadow cell, i.e. it is “born,” then we check whether there is a parent
-
child relationship. This is done by finding the
highest p
parent
-
child
among all eligible ce
lls (i.e. the cells in frame i that were assigned to “die”)
–
if this is above a
threshold then we make the lineage assignment. Full details of the model architecture
,
training
, hyperparameter
optimization
, and post processing
are described in the
S
uppleme
ntal
I
nformation.
Dataset
annotation
and cell segmentation
To power our deep learning approach to cell tracking, we generated a
n
annotated dataset specific to live
-
cell imaging.
This dataset consists of movies of
4
different cell lines
–
HeLa
-
S3, RAW 264.7
, HEK293,
and
NIH
-
3T3. For each cell
line, we collected
fluorescence images
of the cell nucleus
.
We note that the nucleus is a
commonly
used landmark for
quantitative analysis of live
-
cell imaging data
, and that
recent
work has
made it possible to
translat
e brightfield
images into images of
the cell nucleus
20
.
The annot
ations we sought to create consisted
of label movies
–
movies in
which every pixel that belongs to a cell gets a unique integer id in every frame that cell exists
–
and lineage
information which accounts for cell divisions. This latter piece of information
, referred to as relational data, ta
kes
the form of a JSON object that links the ids of parent cells with the ids of child cells. In total, our dataset consists of
11,393
cell trajectories (~
25
frames per trajectory) with
855
cell divisions in total. This
dataset is
as
essential to our
approach as the deep learning code itself. Existing single
-
cell datasets were not adequate for a deep learning
approach, as they were either too small
35
or did not
contain temporal information
13,36
.
Our approach to constructing this dataset is shown in Figure
2a
. Briefly, our dataset annotation consisted of two
phases. The first phase relied on crowdsourcing and internal annotators. Using the Figure 8 platfo
rm, annotators
were given a seq
uence of frames and instructed to color each cell with a unique color for every frame it appeared in.
In this fashion, contributors provided both segmentation and tracking annotations simultaneously. Internal
annotators took
these annotations and manually
corrected errors and recorded cell division events in these data.
Once enough training data was collected (~2,000 trajectories), we trained
preliminary
models for segmentation and
cell tracking. These were accurate enough
to
empower
annotators to correct
a
lgorithm
mistakes as opposed to
creating annotations from scratch. To facilitate this human
-
in
-
the
-
loop approach, we developed a software tool
called Caliban
37
to specifically
curate live
-
cell imaging data.
Caliban,
also
shown in Figure
2a
, takes in segmented and
tracked live
-
cell imaging data and enables
users to
quickly
correct errors using their keyboard and mouse.
The resulting dataset was used to train our
cell tracking
m
odel as well as nuclear segment
ation models. We used a
model based on RetinaMask
38
for nuclear segmentation
, which p
rovided moderate gains to our p
reviously published
approach
13
(
Table S1
)
.
We also
used our pipeline for crowdsourcing to create single cell annotations of
st
atic
fluorescent cytoplasm images and bright
field
images of
7
different cell lines
–
MSC
(mesenchymal stem cells)
,
NIH
-
3T3,
A549, HeLa, HeLa
-
S3, CHO, and
PC3.
Thi
s dataset consiste
d of
63,280
single cell annotations and
was
used to
train models for single
cell segmentation of fluorescent
cytoplasm and brightfield images
, allowing us
to benchmark
our cell tracking algorithm on cytoplasmic images.
Full details of
our annotation method
s
, model architectures,
and
model training can be found in the Suppleme
ntal
Information.
Deployment
The need for hardware acceleration and software infrastructure can pose a significant barrier to entry for new
adopters of deep learning methods
15
. This is particularly true for large inference tasks, as is often the case in
the life
sciences. To solve these issues, we recently developed a platform for performing large
-
scale cellular image analysis
in the cloud using deep learning
-
enabled
models
39
. This platform uses a micro
-
service architecture and allows
sequences of image analysis steps
–
some enabled by deep learning and
some not
-
to be applied to an image before
returning the result to the user. It also sc
ales resources requested from cloud computing providers to meet demand.
.
CC-BY-NC-ND 4.0 International license
It is made available under a
(which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
The copyright holder for this preprint
.
http://dx.doi.org/10.1101/803205
doi:
bioRxiv preprint first posted online Oct. 13, 2019;
5
This ability allows large analysis tasks to be finished quickly, making data transfer the sole b
ottleneck.
Further, this
software allows
analysis to be performed through a web portal
.
We integrated our deep learning
-
enabled cell segmentation and tracking software into this platform
(Figure 2
b)
,
allowing
users to interface with this algorithm through
a web portal. Data in the form of a tiff stack (or a zip file with
directories that cont
ain multiple tiff stacks) is uploaded into a cloud bucket. Once there, the images are segmented,
cells are tracked, and the end result is returned to the user in the fo
rm of “.trk” files, a custom file format with the
raw movie, the label movie, and a json
with the mother
-
daughter information. The results can be curated with
Caliban to correct errors and then queried for single cell analysis using user generated scripts.
While we have made
this work is accessible in the form of Jupyter notebooks for both tr
aining and inference, incorporating this algorithm
into a cloud deployment
makes
it more accessible as analysis can be performed through a web portal. Further, it
will
make it significantly easier to
perform
large inference
tasks
.
Benchmarking
A visual m
ontage of our algorithm’s output is shown in Figure 3a.
To benchmark our method, we reserved
1
0
% of
our annotated data
solely
for testing. We also made use of the ISBI cell tracking dataset; where necessary, we used
our pipeline to create label mov
ies of t
hese data.
As a baseline for the current state
-
of
-
the
-
art, we used an existing
implementation of
the Viterbi algorithm
29,40
.
One challenge of be
nchmarking tracking methods is that errors can
arise from both segmentation
and tracking. Here, we make use of three differen
t tracking metrics.
The first are
confusion matrice
s for our deep learning model
, which
provides a sense of
which linkage errors ar
e most likely
.
The
second
is a graph
-
based metric
35,41
that treats
cell
lineage as a directed acyclic graph (DAG). The number of graph
operations (split/delete/add a node and delete/add/change an edge) needed to map t
h
e DAGs generated by an
algorithm to the ground truth DAGs is used to generate
a score from 0 to 1
.
Last,
we
quantified the
true positive, false
positive, and false negative rates for detecting cell divisions, one of the most challenging
tasks
of cell trac
k
ing.
We first computed confusion matrices
for our method on our testing dataset; these are shown in
the Supplemental
Information
. These
demonstrate that
the most common
error made by our method is confusing
linkages between
the same cell with
linkages bet
ween mother and daughter cells.
This leads to false division events, wh
ere the mother
cell only has one daughter, and missed cell divisions. The former can be mitigated with appropriate post
-
processing.
Next,
we used the
graph
-
based metric
to c
ompare the p
erformance of our method to a Viterbi based method that has
produced state
-
of
-
the
-
art performance
in the ISBI cell tracking challenge
; this is shown in Figure 3b
. To
separate cell
segmentation performance from
cell tracking performance, we appl
ied this met
ric
in three settings. First
,
we used a
classical computer vision
method
to segment cells
and applied the Viterbi cell tracking algorithm to generate a
baseline score.
Next, t
o measure the improvement provided by deep learning
-
enabled cell segm
entation,
we
used
deep learning
to generate cell segmentations and applied both
the Viterbi and our method to
link
cells
together over
time
. Last, to
measure the
improvement provided by deep learning
-
enabled cell
tracking, we used our ground truth
segmentations as the
input to both cell trackers.
This comparison
(Figure
3b
)
reveals that the bulk of the
performance boost of
fered by deep learning comes from improved cell segmentations, an insight that is consistent
with previous work
13
.
This comparison also shows that o
ur deep learning
-
enabled cell tracker
outperforms the
Viterbi algorithm
on these
data
with respect to
the graph
-
based metric, albeit
by a small margin
.
While
the graph
-
based metric
provides a global
measure of performance, it
is less informative for
rare but important
events like cell divisions.
To complete our analysis,
we quantified the
recall, precision, and F1 score
for cell division
detection
on our held
-
o
ut datasets
. This was done for both deep learning generated and ground truth segmentations
for
our method and the Viterbi algorithm. As seen in Figure
3c
, deep learn
ing provides a
marked improvement in cell
division detection, and hence lineage constructio
n.
With ground truth segmentations
, our approach
achieves
a recall
and precision of 89% and 84%
;
the Viterbi algorithm
achieves 57% and 85% respectively on these mea
sures
.
The
performance of our method falls to a recall of 72% and a precision of 71% when d
eep learning generated
segmentations are used.
These results are consistent with the m
inor differences seen in the graph
-
based metric
because divisions are rare even
ts
, and hence require only a few graph operations to
fix if they are misidentified.
This
.
CC-BY-NC-ND 4.0 International license
It is made available under a
(which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
The copyright holder for this preprint
.
http://dx.doi.org/10.1101/803205
doi:
bioRxiv preprint first posted online Oct. 13, 2019;
6
an
alysis highlights the strength of our approach;
live
-
cell imaging experiments that
require
correct lineage
construction stand to benefit the most.
Last
,
because our deep learning model was trained in
the same fashion as
Siamese neural networks (i.e. same v
s
different),
we wondered whether
it
would
generalize beyond just nuclear data.
To test this, we used
our
crowdsourcing pipeline to generate label movies
of
brightfield
and fluorescent
cytoplasmic data
from the ISBI cell
tracking challenge. We then used th
e segmentations from these label movies as
the input into our cell tracker.
Surprisingly
, our cell tracker
performed
markedly well on this challenge, des
pite never having seen cytoplasmic data.
This finding means that single model can be used to track cell
s
irrespective of th
e imaging modality.
This raises the
possibility of a pipeline that can process live
-
cell imaging data
while being agnostic to image t
ype or
acquisition
parameters. As a proof of principle,
w
e constructed
a pipeline that uses
deep learni
ng models to
find the relative
scale of input images to our training data and
identify the imag
ing modality. Using this information, we
can
rescale
images, direct them to the appropriate segmentation model, and
then send the results to our deep learning
-
ba
sed
cell tracker for lineage construction.
While
this demonstrates the feasibility of analyzing
diverse datasets with a
single pipeline
,
additional
training
data
is
necessary
to produce
cytoplasmic segmentations accura
te enough
for
automated analysis
.
Disc
ussion
Live
-
cell imaging is a transformative technology
.
I
t has led to numerous insights into the behavior of single
cells and
will continue to do so for many years to come as imaging and reporter technologies evolve.
While the adoption of
this method has
typically been limited to labs with the computational
expe
rtis
e single cell analysis of these data
demands
,
the arrival of deep learning
is changing this landscape.
With suitable architectures and deployment tools,
deep learning
can turn
single cell image
segmentation
in
to a data annotation problem. With the work
we present
here, the same can be said for cell tracking.
The applications of this
technology
are numerous
, as it
enables high
throughput studies of cell signaling, cell lineage experiments, and
pot
entially
dynamic
live
-
cell
imaging
-
based
screens of pharma
ceutical compounds.
While deep learning methods are powerful, they are not without limitations. Th
is is particularly true for our
approach. Accurate detection is still essential to cell tracking per
formance
, and deep learning
-
based segmentation
methods sti
ll make impactful errors as cells become more crowded.
We expect this to
be
mitigated as more expansive
sets of data are annotated, and as segmentation methods that use spatiotemporal information to
inform
segmentation decisions come online
42
.
This method, and all supervised machine learning methods, is limite
d by the
training data that empower it.
Ou
r training dataset contains less than 1000 cell divisions;
we expect
division
detection
to
become more accurate with additional annotated data.
Because our training data did not include
perturbations
that
markedly
change cell phenotypes
or fates
(i.e.
apop
tosis or
differentiation
)
,
it is possible
performance will be limited
if these
are features of processed data
.
This
can
be mitigated by collecting additional
training data
; we anticipate our existing model
s
combine
d with
a human
-
in
-
the
-
loop approach will
enhance
future
annotation efforts.
We also focused on 2D images, as are collected
with widefield imaging. Modern confocal and light
sheet microscopes can collect 3D data over long time periods
. We suspect that our a
pproac
h
can be adapted to these
data
by using 3D
deep learning sub
-
models
, but the
requisite
annotation tas
k is
more challenging
than the one
undertaken here
.
Lastly, our work has centered on live
-
cell imaging of
mammalian cell lines. While these
are impo
rtant
model systems
for understanding human biology,
and the potential
of deep learning applications to these systems has
for
improving
human health is
substantial,
they
vastly
under sample
the diversity of life.
Much of our understanding of living
systems
comes
from
basic science explorations of bacteria, yeast,
and
archaea.
Live
-
cell imaging and single cell
analysis are powerful methods
for these systems
;
extending
deep learning
-
enabled
method
s
to these systems
43
by
annotating the requisite data could
be just as impactful, if not more so,
as the discoveries
that will be derived from
the
work
presented
here.
.
CC-BY-NC-ND 4.0 International license
It is made available under a
(which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
The copyright holder for this preprint
.
http://dx.doi.org/10.1101/803205
doi:
bioRxiv preprint first posted online Oct. 13, 2019;
7
Acknowledgements
We thank Anima Anandkumar,
Michael Angelo,
Michael Elowitz,
Christopher Frick,
Lea Geontoro,
Kerwyn Casey
Huang,
and Gregory Johnson for helpful suggestions
and sharing data
. We thank Ian Brown
and
Andy Butkovic for
assistance us
ing the Figure 8 image annotation platform, as well as numerous anonymous annotators whose efforts
enabled this work.
We also thank Henrietta Lacks for graciously
donating source mate
rial.
We gratefully
acknowledge support from the
Paul
Allen
Family Founda
tion through the
Discovery Center
s
at Stanford University
and Caltech
,
The Rosen Center
for
Bioengineering
at Caltech,
The Center for Environmental and Microbial
Interactions at Calte
ch,
Google Research Cloud, Figure 8’s AI for everyone award, and a subawa
rd from NIH
U24CA224309
-
01.
Author contributions
EM
, WG, and
DVV conceived of the project
;
EM, EB, MS, DB
, WG, and DVV
designed
and wrote
the
cell tracking
algorithm
and its
deployment
;
EB, EM, and GM designed and wrote the Caliban software
; GM designed a
nd oversaw
the data annotation;
GM, NK, IC
,
DK, CP,
and TP annotated data;
MS, EM, and CP
designed and performed
benchmarking;
TK and EP collected data for annotation;
EM and
DVV wrote the paper; DVV supervised the project.
Datasets
All of the data used in
this paper and the associated annotations can be accessed at
http://www.deepcell.org/data
or at
http://www.github.com/vanvalenlab
through the datasets
module.
S
ource code
A persistent deployment of the software described here can be accessed at
http://www.deepcell.org
. All source
code for cell tracking is available in the DeepCell repository at
http://www.github.com/vanvalenlab
/deepcell
-
tf
.
The source code for the Caliban
software
is available at
http://www.github.com/vanvalenlab/Caliban
. Detailed
instructi
ons are available at
http://deepcell.readthedocs.io/
.
Competing interests
The authors have
filed a provisional patent for the described work; the software described here is available under a
modified Apache
license and is free for non
-
commercial uses.
.
CC-BY-NC-ND 4.0 International license
It is made available under a
(which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
The copyright holder for this preprint
.
http://dx.doi.org/10.1101/803205
doi:
bioRxiv preprint first posted online Oct. 13, 2019;
8
References
1.
Pur
vis, J. E. & Lahav, G. Encoding and Decoding Cellular Information through Signaling Dynamics.
Cell
152
,
945
–
956 (2
013).
2.
Selimkhanov, J.
et al.
Accurate information transmission through dynamic biochemical signaling networks.
Science
346
, 1370
–
1373 (2014
).
3.
Regot, S., Hughey, J. J., Bajar, B. T., Carrasco, S. & Covert, M. W. High
-
sensitivity measurements of multip
le
kinase activities in live single cells.
Cell
157
, 1724
–
1734 (2014).
4.
Elowitz, M. B., Levine, A. J., Siggia, E. D. & Swain, P. S. Stochast
ic Gene Expression in a Single Cell.
Science
297
,
1183
–
1186 (2002).
5.
Golding, I., Paulsson, J., Zawilski, S. M.
& Cox, E. C. Real
-
Time Kinetics of Gene Activity in Individual Bacteria.
Cell
123
, 1025
–
1036 (2005).
6.
Weinberger, L. S., Burnett, J. C., Toe
ttcher, J. E., Arkin, A. P. & Schaffer, D. V. Stochastic Gene Expression in a
Lentiviral Positive
-
Feedback Loop: H
IV
-
1 Tat Fluctuations Drive Phenotypic Diversity.
Cell
122
, 169
–
182
(2005).
7.
Bintu, L.
et al.
Dynamics of epigenetic regulation at the singl
e
-
cell level.
Science
351
, 720
–
724 (2016).
8.
Lane, K.
et al.
Measuring Signaling and RNA
-
Seq in the Same Cell Lin
ks Gene Expression to Dynamic Patterns of
NF
-
κB Activation.
Cell Syst.
4
, 458
-
469.e5 (2017).
9.
Hormoz, S.
et al.
Inferring Cell
-
State Transition Dynamics from Lineage Trees and Endpoint Single
-
Cell
Meas
urements.
Cell Syst.
3
, 419
-
433.e8 (2016).
10.
Forema
n, R. & Wollman, R. Mammalian gene expression variability is explained by underlying cell state.
bioRxiv
626424 (2019) doi:10.1101/626424.
11.
Girkin, J. M. & Carvalho, M. T. The light
-
sheet microscopy r
evolution.
J. Opt.
20
, 053002 (2018).
12.
Ni, Q., Meh
ta, S. & Zhang, J. Live
-
cell imaging of cell signaling using genetically encoded fluorescent reporters.
FEBS J.
285
, 203
–
219 (2018).
13.
Van Valen, D. A.
et al.
Deep Learning Automates the Quantitative A
nalysis of Individual Cells in Live
-
Cell
Imaging Expe
riments.
PLOS Comput. Biol.
12
, e1005177 (2016).
14.
LeCun, Y., Bengio, Y. & Hinton, G. Deep learning.
Nature
521
, 436
–
444 (2015).
15.
Moen, E.
et al.
Deep learning for cellular image analysis.
Nat. Meth
ods
1 (2019) doi:10.1038/s41592
-
019
-
0403
-
1.
16.
Belth
angady, C. & Royer, L. A. Applications, promises, and pitfalls of deep learning for fluorescence image
reconstruction.
Nat. Methods
1 (2019) doi:10.1038/s41592
-
019
-
0458
-
z.
17.
Kraus, O. Z.
et al.
Automat
ed analysis of high‐content microscopy data with deep
learning.
Mol. Syst. Biol.
13
,
924 (2017).
18.
Weigert, M.
et al.
Content
-
Aware Image Restoration: Pushing the Limits of Fluorescence Microscopy. (2018).
19.
Christiansen, E. M.
et al.
In Silico Labelin
g: Predicting Fluorescent Labels in Unlabeled Images.
Cell
173
, 792
-
803.e19 (2018).
20.
Ounkomol, C., Seshamani, S., Maleckar, M. M., Collman, F. & Johnson, G. R. Label
-
free prediction of three
-
dimensional fluorescence images from transmitted
-
light microsc
opy.
Nat. Methods
1 (2018)
doi:10.1038/s41592
-
018
-
0111
-
2.
21.
Ronneberger, O., Fischer, P. & Brox, T. U
-
Net: Convolutional Networks for Biomedical Image Segmentation. in
Medical Image Computing and Computer
-
Assisted Intervention
–
MICCAI 2015
(eds. Navab,
N., Hornegger, J.,
Wells, W. M. & Frangi, A. F.) 234
–
241 (Springer International Publishing, 2015).
22.
Falk, T.
et al.
U
-
Net: deep learning for cell counting, detection, and morphometry.
Nat. Methods
16
, 67 (2019).
23.
Keren, L.
et al.
A Structured Tumor
-
Immune Microenvironment in Triple Negative Breast Cancer Revealed by
Multiplexed Ion Beam Imaging.
Cell
174
, 1373
-
1387.e19 (2018).
24.
Haberl, M. G.
et al.
CDeep3M
—
Plug
-
and
-
Play cloud
-
based deep learning for image segmentation.
Nat. Methods
15
, 677
–
680 (20
18).
25.
Akram, S. U., Kannala, J., Eklund, L. & Heikkilä, J. Cell tracking via proposal generation and selection. (2017).
26.
Tsai, H.
-
F., Gajda, J., Sloan, T. F. W., Rares, A. & Shen, A. Q. Usiigaci: Instance
-
aware cell tracking in stain
-
free
phase
contr
ast microscopy enabled by machine learning. (2019).
27.
Newby, J. M., Schaefer, A. M., Lee, P. T., Forest, M. G. & Lai, S. K. Convolutional neural networks automate
detection for tracking of submicron
-
scale particles in 2D and 3D.
Proc. Natl. Acad. Sc
i.
11
5
, 9026
–
9031 (2018).
.
CC-BY-NC-ND 4.0 International license
It is made available under a
(which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
The copyright holder for this preprint
.
http://dx.doi.org/10.1101/803205
doi:
bioRxiv preprint first posted online Oct. 13, 2019;
9
28.
Kudo, T.
et al.
Live
-
cell measurements of kinase activity in single cells using translocation reporters.
Nat.
Protoc.
13
, 155
–
169 (2018).
29.
Magnusson, K. E. G., Jalden, J., Gilbert, P. M. & Blau, H. M. Global linking of cell trac
ks using the Viterbi algorithm.
IEEE Trans. Med. Imaging
34
, 911
–
929 (2015).
30.
Payer, C., Štern, D., Neff, T., Bischof, H. & Urschler, M. Instance Segmentation and T
racking with Cosine
Embeddings and Recurrent Hourglass Networks. in
Medical Image Computi
ng and Computer Assisted
Intervention
–
MICCAI 2018
(eds. Frangi, A. F., Schnabel, J. A., Davatzikos, C., Alberola
-
López, C. & Fichtinger, G.)
3
–
11 (Springer Internati
onal Publishing, 2018).
31.
Bannon, D.
et al.
Dynamic allocation of computational resourc
es for deep learning
-
enabled cellular image
analysis with Kubernetes.
bioRxiv
505032 (2019) doi:10.1101/505032.
32.
Jaqaman, K.
et al.
Robust single
-
particle tracking
in live
-
cell time
-
lapse sequences.
Nat. Methods
5
, 695
–
702
(2008).
33.
Sadeghian, A., Ala
hi, A. & Savarese, S. Tracking The Untrackable: Learning To Track Multiple Cues with Long
-
Term Dependencies. in
Computer Vision and Pattern Recognition (CVPR)
(2017).
34.
Kuhn, H. W. The Hungarian method for the assignment problem.
Nav. Res. Logist. Q.
2
,
83
–
97 (1955).
35.
Maška, M.
et al.
A benchmark for comparison of cell tracking algorithms.
Bioinformatics
30
, 1609
–
1617 (2014).
36.
Caicedo, J. C.
et al.
Evaluation of
Deep Learning Strategies for Nucleus Segmentation in Fluorescence Images.
(2018).
37.
Cl
aremont, C. & Cockrum. Uncanny X
-
Men.
38.
Fu, C.
-
Y., Shvets, M. & Berg, A. C. RetinaMask: Learning to predict masks improves state
-
of
-
the
-
art single
-
shot
detection for
free.
ArXiv190103353 Cs
(2019).
39.
Bannon, D.
et al.
DeepCell 2.0: Automated cloud depl
oyment of deep learning models for large
-
scale cellular
image analysis. (2018).
40.
Ulman, V.
et al.
An Objective Comparison of Cell Tracking Algorithms.
Nat. Methods
14
, 1141
–
1152 (2017).
41.
Matula, P.
et al.
Cell Tracking Accuracy Measurement Based on C
omparison of Acyclic Oriented Graphs.
PLOS
ONE
10
, e0144959 (2015).
42.
Voigtlaender, P.
et al.
MOTS: Multi
-
Object Tracking and Segmentation. in 7942
–
7951 (2019).
43.
Lugagne, J.
-
B., Lin, H. & Dunlop, M. J. DeLTA: Automated cell segmentation, tracking, and
lineage reconstruction
using deep learning.
bioRxiv
720615 (2019) doi:10.1101/720615.
44.
Lin, T., Goyal, P., Girshick, R., He, K. & Dollar, P. Focal Loss for Dense O
bject Detection. in
2017 IEEE
International Conference on Computer Vision (ICCV)
2999
–
300
7 (2018). doi:10.1109/ICCV.2017.324.
45.
Ren, S., He, K., Girshick, R. & Sun, J. Faster R
-
CNN: Towards Real
-
Time Object Detection with Region Proposal
Networks. in
Adv
ances in Neural Information Processing Systems 28
(eds. Cortes, C., Lawrence, N. D., Lee,
D. D.,
Sugiyama, M. & Garnett, R.) 91
–
99 (Curran Associates, Inc., 2015).
46.
Kirillov, A., Girshick, R., He, K. & Dollar, P. Panoptic Feature Pyramid Networks. in 63
99
–
6408 (2019).
47.
Kingma, D. P. & Ba, J. Adam: A Method for Stochastic Optimization.
Ar
Xiv14126980 Cs
(2014).
48.
Walt, S. van der
et al.
scikit
-
image: image processing in Python.
PeerJ
2
, e453 (2014).
.
CC-BY-NC-ND 4.0 International license
It is made available under a
(which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
The copyright holder for this preprint
.
http://dx.doi.org/10.1101/803205
doi:
bioRxiv preprint first posted online Oct. 13, 2019;
10
Figures
Figure 1
:
Tracking single cells
with deep learning and linear programming. (a
, b
)
Computational analysis is a
significant barrie
r for extracting single cell information from movies of living cells.
Cells
must be identified in every
frame and then these detections must be linked together
over time to form a temporal record for each cell.
(c)
Cell
tracking can be framed as a linear a
ssignment problem in which N
i
objects in frame i are matched up with N
i+1
objects
in frame i+1. Shadow objects can be introduced to account for births (i.e. fro
m cell division events) or deaths (i.e.
cells leaving the field of view). Solving the linear ass
ignment problem requires first creating a cost matrix that scores
each possible assignment. The Hungarian algorithm
34
is then used to find the optimal assignment that minimizes the
cost function. Instead of manually engineering this cost function, we use a deep lear
ning model to learn one from
annotated data. Here, p
same
is the probability that two cells being
compared are the same, b is the cost associated
with cell “births” (i.e. a cell in frame i+1 being assigned to a shadow cell), and d is the cost associate with
a cell death
(i.e. a cell in frame i being assigned to a shadow cell). A deep learning model l
earns to take information from two
cells and
compute
the probability these are the same cell, different cells, or have a parent
-
child relationship. This
model t
akes information on each cell’s appearance,
local neighborhood,
morphology,
and
motion and summa
rizes
as a vector using
a deep learning sub
-
model. A fully connected layer reads these summaries and determines the
scores for the three classes.
.
CC-BY-NC-ND 4.0 International license
It is made available under a
(which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
The copyright holder for this preprint
.
http://dx.doi.org/10.1101/803205
doi:
bioRxiv preprint first posted online Oct. 13, 2019;
11
Figure
2
:
A h
uman
-
in
-
the
-
loop
to dataset construction
and
cloud computing facilitate a scalable solution to
live
-
cell
image analysis.
(
a
)
Combining crowd sourcing and a human
-
in
-
the
-
loop approach to dataset annotation enables the
construction of an ImageNet for live
-
cell imaging.
By annotating montages, crowd contributors both segment and
track single cells in
live
-
cell imaging data.
This
data leads to models that are used to process additional data; e
xpert
annotators
use Caliban to
correct
model
errors and
identify
cell division events. The resulting data is then used to
train a
final set of
deep learning model
s
to perform cell segmentation and tracking.
(b) Integration of a cell tracking
service into DeepCell 2.0. Datasets are uploaded to a cloud bucket; once there,
a tracking consumer object facilitates
interactions with deep learning
models
via Tensorflow ser
ving
for segmentation and tracking. The implementation
within the Kubernetes engine includes an autoscaling module that monitors resource utilization and scale
s the
compute resources accordingly.
.
CC-BY-NC-ND 4.0 International license
It is made available under a
(which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
The copyright holder for this preprint
.
http://dx.doi.org/10.1101/803205
doi:
bioRxiv preprint first posted online Oct. 13, 2019;
12
Figure
3
:
Benchmarking demonstrates that deep learning a
chieves state
-
of
-
the
-
art performance on cell tracking
tasks for a variety of cell types. (a) A montage of tracking results for fluorescent images of cell nuclei and brightfield
images of cells. (b) Confusion ma
trices
for our deep learning model
identify th
e linkages between mother cells and
daughter cells as
our dominant error mode
.
These linkage errors lead to
erroneous and missed divisions.
(c)
A graph
-
based metric for cell tracking demonstrates
that deep lea
rning
enables
state
-
of
-
the
-
art performance
, wi
th the bulk
of this performance boost coming from
improved segmentations. (d) Analysis of performance in cell division
detection reveals that the performance boost offered by deep learning comes from more accur
ate detection of cell
divisions.
.
CC-BY-NC-ND 4.0 International license
It is made available under a
(which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
The copyright holder for this preprint
.
http://dx.doi.org/10.1101/803205
doi:
bioRxiv preprint first posted online Oct. 13, 2019;
13
Supplemental
Information
Cell line acquisition
and culture methods.
We used the mammalian cell lines NIH
-
3T3, HeLa
-
S3, HEK 293, and RAW
264.7 to collect training data for
nuclear segmentation and the cell line
s
NIH
-
3T3 and
RAW 264.7 to collect training
data for augmen
ted microscopy.
All cell lines were acquired from ATCC. The cells have not been authenticated and
were not tested for mycoplasma contamination.
Mammalian cells were cultured in Dulbecco’s modified E
agle’s medium (DMEM, Invitrogen
or Caisson
)
supplemented
with 2mM L
-
Glutamine (Gibco), 100 U/ml penicillin, 100μg/ml streptomycin (Gibco
or Caisson
), and
either 10% fetal bovine serum (Omega Scientific
or Thermo Fisher
) for HeLa
-
S3 cells, or 10% calf serum
(Colorado
Serum Company) for NIH
-
3T3 cells.
Cells were
incubated at 37°C in a humidified 5% CO2 atmosphere. When 70
-
80% confluent, cells were passaged and seeded onto fibronectin coated glass bottom 96
-
well plates (Thermo Fisher)
at 10,000
-
20,000 cells/
well. The seeded cells were then incubated for 1
-
2 hours
to allow for cell adhesion to the
bottom of the well plate before imaging
.
Collection of live
-
cell imaging data
.
For
fluorescent nuclear
imaging, mammalian cells were seeded onto fibronectin
(Sigma,
10ug/ml) coated glass bottom 96
-
well plates (Nunc) and al
lowed to attach overnight. Media was removed
and replaced with imaging media (FluoroBrite DMEM (Invitrogen) supplemented with 10mM Hepes, 1% FBS, 2mM
L
-
Glutamine) at least 1 hour prior to imaging
.
Fo
r nuclear imaging, c
ells without a genetically encoded nu
clear
marker were incubated with 50ng/ml Hoechst (Sigma) prior to imaging.
For cytoplasm imaging, cells were incubated
with
2 μM
CellTracker CMFDA
prior to imaging.
Cells were imaged with
either
a Nikon Ti
-
E
or Nikon Ti2
fluorescence
microscope with enviro
nmental control (37°C, 5% CO
2
) and controlled by Micro
-
Manager
or Nikon Elements
. Images
were acquired with a 20x objective (40x for RAW 264.7 cells) and
either
a
n
A
ndor Neo 5.5 CMOS camera with 2x2
binning
or a Photometrics Prime 95B CMOS camera with 2x2
binning. All data was scaled to so that pixels had the
same physical dimension prior to training.
Fluorescence images were taken for nuclear
data
, while both brightf
ield
and fluorescence im
ages were taken for cytoplasmic data.
For time
-
lapse experiments, i
mages were acquired at 6
-
minute intervals.
Deep learning architecture for single
-
cell segmentation
.
Our pipeline for single cell segmentation
is shown in
Figure
S
1
. This pipeline
uses deep learning models to rescale images
and direct them to the appropriat
e segme
ntation.
We
used modified RetinaMask
38
model
s
for
single cell
segmentation
of fluorescent nuclear, fluorescen
t cytoplasm, and
brightfield images
. RetinaMask generates instance masks in a fashion similar to Mask
-
RCNN but uses single shot
detection like RetinaNet
44
rather than feature proposals
45
to identify objects.
Each model used a ResNet50
backbone pre
-
trained on ImageNet. For nuclear segmentation, we used the P3
and P4
feature pyramid layers for
object
detection with
an anchor size of 16
and 32
pixels
respectively
.
For the fluorescent cytoplasmic and
brightfield segmentation, we used the P3, P4, P5, and
P6 layers with anchor sizes of 32, 64, 128, and 256 pixels. For
all three models, we attached two sem
antic segmentation heads
46
to predic
t pixelwise and deep watershed
segmentations. This encouraged the backbone and feature pyramid network to learn more general image features.
We used a wei
ghted softmax loss for both heads that was weighted by 0.1. All three models were trained on their
res
pective datasets in the same fashion. We used the
Adam
47
optimization algorithm with a learning rate of 10
-
5
and c
lip
norm of 0.001, batch size of 4, and
L2 regularization strength of 10
-
5
for 16 epochs on a NVIDIA V100
graphics card.
For the nuclear data, we ensured that
our training/validation split was the same as was used for
training the cell tracking model.
Our
post processing
consisted of removing
segmentation
masks that have high
ove
rlap with >2 other masks.
Masks that only overlapped with 1 other mask
were resolved using a marker based
random walker segmentation step
48
. All
masks
smaller than 100 pixels were
also
removed
during post processing.
For nuclear segmentation, we used the out
put of the watershed se
mantic segmentation mask to add cells that were
missed by the RetinaMask object detection.
.
CC-BY-NC-ND 4.0 International license
It is made available under a
(which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
The copyright holder for this preprint
.
http://dx.doi.org/10.1101/803205
doi:
bioRxiv preprint first posted online Oct. 13, 2019;
14
Deep learning
models
for
scale and
image type detection.
To develop a live
-
cell imaging analysis workflow that is
agnostic to imaging modali
ty and acquisition para
meters
, we trained two deep learning models for
detect
ing
scale
and image type
.
The scale detection model
sought to identify the relative scale of between an input image and our
training data; we
applied
affine transformations
to our
training data to creat
e images of different scales. The model
consisted of a MobileNetV2 backbone
connected to an average pooling layer and followed by two dense layers
. The
model was trained
for 20 epochs
on a combined dataset (nuclear, fluorescent cytop
lasmic, and brightfield
images)
using
a mean squared error (MSE) loss. We used
the Adam optimization algorithm
with a learning rate of 10
-
5
and
clip norm of 0.001, batch size of 64, and
L2 regularization strength of 10
-
5
on an NVIDIA V100 graphics card. Th
e
image type detection model consisted of a MobileNetV2 backbone connected to an average pooling
layer
followed
by two dense layers and a softmax layer. The model was trained for 20 epochs on
a combined dataset using a
w
eighted categorical crossentropy los
s. We used We used the Adam optimization algorithm with a learning rate of
10
-
5
and clip norm of 0.001, batch size of 64, and
L2 regularization strength of 10
-
5
on an NVIDIA V100 graphics card.
The scale detection model
achieved a
mean absolute percentage
error of 0.85% on validation data while the image
type detection achieved a classification accuracy of
9
8
% on validation data. We found that using
the MobileNetV2
backbone provided similar performance to larger networks
while offering a higher inference sp
eed and lower
memory footprint.
Figure S
1
Computational pipeline for single cell segmentation. Our pipeline uses a scale detection deep learning
model
to rescale input images to the same physical pixel dimensions of o
ur training data. A
nother deep learn
ing
model detects whether the rescaled images are fluorescent nuclear images, fluorescent cytoplasm images, or
brightfield images.
Once the image type is determined, the images are sent to a RetinaMask based deep learning
model for single cell segmentation
. The segmen
tation masks are then sent to
the cell tracking deep learning model
to construct cell lineages.
.
CC-BY-NC-ND 4.0 International license
It is made available under a
(which was not peer-reviewed) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity.
The copyright holder for this preprint
.
http://dx.doi.org/10.1101/803205
doi:
bioRxiv preprint first posted online Oct. 13, 2019;