of 53
Integrated neurophotonics: Towards dense volumetric
interrogation of brain circuit activity – at depth and in real time
Laurent C. Moreaux
1,‡
,
Dimitri Yatsenko
2
,
Wesley D. Sacher
3,4,5
,
Jaebin Choi
6
,
Changhyuk
Lee
6,7
,
Nicole J. Kubat
1
,
R. James Cotton
8,12
,
Edward S. Boyden
9
,
Michael Z. Lin
10
,
Lin
Tian
11
,
Andreas S. Tolias
12,2
,
Joyce K.S. Poon
5,13
,
Kenneth L. Shepard
6
,
Michael L.
Roukes
1,3,4,14,‡,§
1.
Division of Physics, Mathematics and Astronomy, California Institute of Technology, Pasadena,
CA 91125 USA
2.
Vathes LLC, Houston, TX 77030
3.
Kavli Nanoscience Institute, California Institute of Technology, Pasadena, CA 91125 USA
4.
Division of Engineering and Applied Science, California Institute of Technology, Pasadena, CA
91125 USA
5.
Max Planck Institute for Microstructure Physics, Halle, Germany
6.
Departments of Electrical Engineering and Biomedical Engineering, Columbia University, New
York, NY 10027
7.
Center for BioMicrosystems, Brain Science Institute, Korea Institute of Science and Technology,
Korea
8.
Shirley Ryan AbilityLab, Northwestern University, Chicago, IL 60611
9.
Howard Hughes Medical Institute; McGovern Institute and Koch Institute, and Departments of
Brain and Cognitive Sciences, Media Arts and Sciences, and Biological Engineering, MIT
10.
Departments of Neurobiology and Bioengineering, Stanford University, Stanford, CA 94305
USA
11.
Department of Biochemistry and Molecular Medicine, University of California, Davis, CA95616
USA
Corresponding authors: moreauxl@caltech.edu, roukes@caltech.edu.
§
Lead contact: roukes@caltech.edu
Author Contributions
MLR conceived of this project; LCM and MLR together adapted it to 1p processes; WDS, LCM, JKSP, and MLR led E-pixel probe
development; JC, CL, and KLS led D-pixel probe development; LCM, WDS, JC, JKSP, KLS, and MLR led
in vitro
and
in vivo
experiments. ESB, MZL, and LT led development of molecular reporters and their characterization; DY developed de-mixing analysis
and carried out computational modeling in collaboration with LCM, RJC, KLS, RJC, AST, and MLR. LCM, DY, NC, KLS, and MLR
wrote the manuscript; all authors provided both contributions and editing. All authors reviewed the final manuscript.
Competing Interests
The authors declare no competing interests.
Publisher's Disclaimer:
This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our
customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of
the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered
which could affect the content, and all legal disclaimers that apply to the journal pertain.
HHS Public Access
Author manuscript
Neuron
. Author manuscript; available in PMC 2021 October 14.
Published in final edited form as:
Neuron
. 2020 October 14; 108(1): 66–92. doi:10.1016/j.neuron.2020.09.043.
Author Manuscript
Author Manuscript
Author Manuscript
Author Manuscript
12.
Center for Neuroscience and Artificial Intelligence and Department of Neuroscience, Baylor
College of Medicine, Houston, TX 77030
13.
Department of Electrical and Computer Engineering, University of Toronto, 10 King’s College
Rd., Toronto, Ontario M5S 3G4, Canada
14.
Division of Biology and Biological Engineering, California Institute of Technology, Pasadena, CA
91125 USA
Abstract
We propose a new paradigm for dense functional imaging of brain activity to surmount the
limitations of present methodologies. We term this approach
integrated neurophotonics;
it
combines recent advances in microchip-based integrated photonic and electronic circuitry with
those from optogenetics. This approach has the potential to enable lens-less functional imaging
from within the brain itself
- to achieve dense, large-scale stimulation and recording of brain
activity with cellular resolution at arbitrary depths. We perform a computational study of several
prototype 3D architectures for implantable probe-array modules that are designed to provide fast
and dense single-cell resolution,
e.g.,
within a 1-mm
3
volume of mouse cortex comprising
~100,000 neurons. We describe progress toward realizing integrated neurophotonic imaging
modules, which can be produced
en masse
with current semiconductor foundry protocols for chip
manufacturing. Implantation of multiple modules can cover extended brain regions.
Massively parallel interrogation of brain activity
“Within the central nervous system, the events in each unit are not so important. We
are more concerned with the interactions of large numbers, and our problem is to
find the way in which such interactions can take place.”
-Edward D.
Adrian (1926)
These final lines from Lord Adrian’s Nobel lecture (
Adrian, 1926
) illustrate the
extraordinary prescience of this researcher who first discovered neuronal spiking. He
anticipated that understanding brain computation is not likely to be achieved only by studies
of individual neurons but, instead, by observing coordinated interactions of neurons and
their collective activity patterns.
Realizing instrumentation to monitor population activity within the brain with single-neuron
resolution is a profoundly difficult challenge; Figure 1 provides a sense of the scale
involved. The slow rate of technological development in neuroscience is elucidated in Figure
2; it charts the evolution of our ability to simultaneously resolve and track the activity of a
multiplicity
of neurons
in vivo,
over the six decades since the invention of whole-cell
recording (
Stevenson and Kording, 2011
). Today, the state-of-the-art permits simultaneous,
full bandwidth recording
in vivo
in awake rodents from multi-shank neural probe modules,
each with up to 1,024 channels (
Rios et al., 2016
;
Shobe et al., 2015
). With implantation of
multiple probes of these types, many thousands of neurons are now being simultaneously
recorded (
Steinmetz et al., 2019
). Although it is unequivocal that these advances open
exciting research frontiers, the number of observable neurons has continued to remain
Moreaux et al.
Page 2
Neuron
. Author manuscript; available in PMC 2021 October 14.
Author Manuscript
Author Manuscript
Author Manuscript
Author Manuscript
comparable to the electrode count. This is consistent with the empirical observation that
multi-site extracellular electrodes yield, on average, just one or two units per site, even with
optimal spike-sorting algorithms (
Marblestone et al., 2013
). At this rate of development,
another 90 years
must elapse before the activity of an entire mouse brain, containing roughly
75 million neurons, will become observable (Figure 2). Clearly, we must significantly
accelerate this rate of development.
This was a central aim of our proposal (
Alivisatos et al., 2012
) that eventually culminated in
the launching of the U.S. BRAIN Initiative (
Bargmann and Newsome, 2014
). Our initial
vision, which still remains true, is that advances in nanotechnology, molecular reporters, and
large-scale integration of semiconductor devices now make it feasible to precipitously
upscale the rate of progress toward massively multiplexed interrogation of brain circuits
(
Alivisatos et al., 2012
).
Here, we focus in more depth on these prospects. Our aim in this Perspective is not solely to
identify ways to increase the total number of neurons that can be recorded from
simultaneously. Instead, we explore the possibility of achieving
dense
recording from within
a targeted tissue volume, to ultimately achieve complete interrogation of local brain circuit
activity. We use the word
interrogation
to denote recording and direct causal manipulation of
a brain circuit’s individual neurons by the application of patterned, deterministic stimulation
with single-neuron resolution. To achieve this, we are pursuing a new approach, which we
term
integrated neurophotonics,
that offers significant potential for accelerating progress
toward Lord Adrian’s vision. This technological path offers the prospect of dense functional
imaging of neuronal activity in highly scattering neural tissue - providing cellular-scale
resolution at
arbitrary
depths in the brain. Our approach is based on implanting an entire
lens-less imaging system
within the brain itself,
by distributing dense arrays of microscale
photonic emitter and detector pixels (hereafter, E- and D-pixels) positioned on a 3D spatial
lattice (
Roukes, 2011
;
Roukes et al., 2016
). These pixel arrays are integrated onto narrow
silicon shanks (needles), which leverage recent advances in silicon-nanoprobe-based
fabrication (
Rios et al., 2016
;
Shobe et al., 2015
;
Steinmetz et al., 2018
). Used with
functional molecular reporters (
Andreoni et al., 2019
;
Chen et al., 2013
;
Lin and Schnitzer,
2016
) and optogenetic actuators (
Boyden, 2011
;
Miesenböck, 2011
), this novel
instrumentation offers the prospect of approaching the interrogation of
all neuronal activity
from within a 1-mm
3
volume (~100,000 neurons in mouse cortex). The approach leverages
recent breakthroughs in molecular reporters that can enable multimodal and multi-physical
sensing (Figure 1), advances in optogenetic actuators that enable optical control of neural
activity, and the genetically encoded delivery of reporters and actuators that provide
specificity of cell type. Further, the methodology is potentially scalable - multiple modules
can be tiled to densely cover extended regions deep within the brain. We anticipate this will
ultimately permit
interrogation,
that is, simultaneous recording and patterned stimulation of
millions of neurons, at arbitrary positions and depths in the brain, to unveil dynamics of
neural networks - with single-cell resolution and specificity of cell type. Like their
contemporary counterparts for highly multiplexed electrophysiology (Ephys), ultranarrow
photonic neural probes perturb brain tissue minimally given their small cross-sections and
passivated surfaces. They impose negligible tissue displacement upon implantation, while
dissipating low power during operation - comparable to today’s active, multi-site Ephys
Moreaux et al.
Page 3
Neuron
. Author manuscript; available in PMC 2021 October 14.
Author Manuscript
Author Manuscript
Author Manuscript
Author Manuscript
probes that also employ complementary metal-oxide-semiconductor (CMOS) technology.
And, importantly, they offer near-term prospects for wide deployment to the neuroscience
research community, as they are mass producible by well-validated semiconductor foundry
(microchip- production factory) methods.
Conventional electrophysiology
Background: intracellular v. extracellular Ephys
Intracellular
or “whole-cell” recording was pioneered by Hamill
et al.
in 1981 (
Hamill et al.,
1981
), and this technique has remained the gold standard in Ephys. Its singular advantages
are its high signal-to-noise ratio (SNR), the ability to directly target specific cells, and the
important ability to deduce
connection strengths
between neurons directly, in what are often
termed “multi-patch”
(i.e.,
multi-electrode) experiments. However, there are significant
technical challenges associated with whole cell recording, particularly
in vivo
(
Hulse et al.,
2016
). These arise from the delicate manipulations involved in patching, that prevent
upscaling the technique to enable simultaneous recording from more than roughly a dozen
neurons, the practical limit achieved by today’s expert practitioners (
Jiang et al., 2015
;
Jiang
et al., 2013
;
Perin et al., 2011
). Recent efforts have focused on the automation of patching
protocols (
Kodandaramaiah et al., 2012
), but these have not yet yielded a substantial
increase in the multiplicity of simultaneous whole-cell recording.
Extracellular
electrophysiological recording, as first pioneered by Lord Adrian, is more
amenable to upscaling. Microwire electrodes in twisted pairs (
McNaughton et al., 1983
)
(“stereotrodes”) and quadruples (“tetrodes”) (
Recce and O’Keefe, 1989
), enable single-unit
recordings from several adjacent neurons (
Gray et al., 1995
) when combined with spike-
sorting protocols such as first developed by Gerstein and Clark in 1964 (
Gerstein and Clark,
1964
). Over the past several decades, arrays of tetrodes have been successfully employed to
observe patterns of activity over distributed regions of the brain (
Wilson and McNaughton,
1993
). However, scaling upward to the regime of thousands of channels has proven
challenging.
Electrical recording: The present state-of-the-art
In 1970, well prior to the development of tetrodes, Wise
et al.
first pioneered use of silicon-
based microfabrication techniques to upscale extracellular recording multiplicity (
Wise et
al., 1970
). Here, by
multiplicity
we mean the number of active neurons that can be
simultaneously resolved and recorded. Massively-multiplexed silicon neural probes have
since been widely validated (
Wise et al., 2008
), commercialized, and upscaled to permit
multiplexing of hundreds of recording sites within an individual probe (
Berenyi et al., 2014
).
However, despite the technology’s potential and the advances made, the multiplicity of
extracellular recording has increased only by a factor of two roughly every seven years over
the past sixty years (Figure 2). As mentioned, silicon neural probes have recently been
upscaled to contain over one thousand simultaneously active, full bandwidth recording
electrodes within a single functional multi-probe module. Here, nanofabrication methods
permitting nanowire interconnects along the shanks enable the attainment of narrow shank
cross sections (
Rios et al., 2016
;
Shobe et al., 2015
;
Steinmetz et al., 2018
) that are
Moreaux et al.
Page 4
Neuron
. Author manuscript; available in PMC 2021 October 14.
Author Manuscript
Author Manuscript
Author Manuscript
Author Manuscript
comparable to the diameter of tetrodes (typically, ~35μm) that permit chronic implantation.
The expectation here is that (cross sectional) size matters.
To evaluate the practical and ultimate limits of the recording multiplicity attainable with
multi-site electrophysiological recording, the physical origin of extracellular currents and
potentials must be considered. Extracellular currents arise from the activity of ion channels
and pumps occurring both at soma and at neural processes. The slow spatiotemporal
evolution of the local field potential (LFP) arises from ion exchange between soma,
processes, synapses, and their extracellular environment - that is, throughout the vast regions
over which neural processes extend. By contrast, rapid local waveforms arising from spiking
are dominated by much faster somatic ion currents. (
Berenyi et al., 2014
;
Buzsaki, 2004
;
Buzsaki et al., 2012
) Accordingly, despite the nomenclature, so-called “local field
potentials” tend to be spatially diffuse in character, while fast transients associated with
action potentials are much more spatially localized. Spike sorting protocols with multi-site
electrodes take advantage of the latter; they permit separation and extraction of the fast
temporal activity of individual “units” (arising from single-neuronal spiking) from the LFP
signals (that arise from complex, ensemble-average, multi-neuronal activity.) Recently, the
effectiveness of spike sorting protocols in various experimental contexts have been assessed
qualitatively (
Marblestone et al., 2013
). This meta-analysis of their empirical limits is
sobering. It’s conclusion is that - while future improvements might, in principal, yield up to
10 neurons per electrode for multi-site probes - experiments today typically yield a
recording multiplicity that is smaller,
i.e.,
much closer to 1:1. In other words, on average,
one electrode is required for every neuron recorded. An important caveat here is that the
activity of the brain region probed matters; regions with sparse activity will yield smaller
multiplicities that can be obtained from regions with less-sparse activity.
This low multiplicity obtained from most multi-site Ephys recordings has important
practical implications, both for the potential
level of scale-up
(multiplexing) that is
attainable, and for the likely
density of neuronal coverage
that is practicable. Scaling up the
number of recording sites with 3D multi-shank architectures was first demonstrated in the
1990’s (
Campbell et al., 1991
;
Hoogerwerf and Wise, 1994
). To minimize perturbation of
brain activity, the consensus is that implantable probes should altogether volumetrically
displace only a fraction of a percent of neural tissue (
Marblestone et al., 2013
). However,
this asserted limit is just an estimate; controlled, direct, and systematic studies have not yet
been carried out.
Najafi and Wise first explored the possibilities for massively upscaling the recording
multiplicity of silicon-based multi-site neural probes for Ephys (
Najafi et al., 1990
). They
consider the mechanical robustness of silicon probes and conclude that, to readily permit
penetration of the pia, shanks 5–10mm long (permitting access to deep brain structures in
small mammals) should maintain a cross-sectional area of ~200μm
2
,
e.g.
20μm width × (at
least) 10μm thickness. Maintaining <1% volumetric exclusion of brain tissue for a 3D neural
probe array constructed from such shanks would then impose a minimum inter-shank pitch
of ~140μm, assuming a square grid architecture. With this configuration, and using the fact
that state-of-the art, low-noise single-unit recording requires an electrode geometric surface
area of ~180μm
2
(~15μm diameter for a circular electrode) (
Ludwig et al., 2011
), the
Moreaux et al.
Page 5
Neuron
. Author manuscript; available in PMC 2021 October 14.
Author Manuscript
Author Manuscript
Author Manuscript
Author Manuscript
maximum practical number of electrodes per unit volume of brain tissue can be estimated.
Assuming each shank comprises a linear array of electrodes with a pitch of 1.5 times the
electrode diameter
(i.e.,
22.5μm center- to-center spacing), a maximum fill-factor of
approximately 2,100 electrodes/mm
3
is attainable. One cubic millimeter of mouse cortex
contains approximately 100,000 neurons and, being (overly) optimistic by assuming that a
maximum of
two
units are extracted per recording site, we conclude that a volume coverage
of only ~4% of the neuronal population is the upper limit of attainable coverage. Again, this
must be qualified that that coverage obtained will be dependent on the relative level of
activity within the brain region probed.
These considerations can be revised somewhat in light of the new generation of ultra-
flexible neural probes emerging from multiple laboratories (
Jeong et al., 2015
;
Liu et al.,
2015
;
Rousche et al., 2001
). Thinner probes - which, for implantation, require
removable
stiffeners
- could, in principle, permit closer probe spacing and, thereby, increase coverage
of the neural population. Current efforts pursuing flexible probe technology either focus on
devices based on polymeric materials or ultra-thinned silicon. The latter make use of
fabrication protocols permitting conventional semiconductors - usually considered brittle,
inelastic materials - to be rendered extremely pliable by making them extremely thin.
Thinning the silicon substrate has the two very significant benefits: it yields much thinner
devices than their polymeric counterparts, and it enables use of state-of-the-art silicon
CMOS circuitry (
Navaraj et al., 2018
;
Shahrjerdi and Bedell, 2013
). A recent analysis for
these new flexible probe designs, in the same vein as that of Najafi and Wise, arrives at
somewhat more optimistic conclusions (
Kleinfeld et al., 2019
).
Ultra-thinned CMOS silicon neural probes can thus be made almost an order of magnitude
thinner than those estimated by Najafi et al. (
Najafi et al., 1990
) - and this could reduce
tissue displacement by a similar factor. Following the logic above, it could permit reducing
shank spacing down to a ~50μm pitch. However, there are significant challenges to
achieving reproducible implantation of such flexible 3D arrays. The requisite
stiffeners
, used
to facilitate probe implantation, will likely displace a comparable amount of tissue as the
somewhat thicker silicon probes that the flexible devices are intended to replace.
Nonetheless, it is interesting to note that a 50μm shank separation would be commensurate
with the empirically deduced range of multi-site electrical recording (
Buzsaki, 2004
). The
limitation on Ephys range arises from the combined biophysical effects of electrostatic
screening in the ionic cerebrospinal fluid, the presence of other distributed sources of
electrophysiological activity (which can raise the practical noise floor for detection) and the
need to maintain sufficient SNR in extracellular recordings to permit efficacious spike-
sorting. Following the logic outlined above to ensure that volume exclusion of neural tissue
kept below 1%, a ten-fold increase in electrode density over the estimate above would permit
volumetric coverage of up to 40% of adjacent neurons with an ultrathin probe array. But we
emphasize that this is an optimistic estimate; it is predicated on implanting an immense
number of flexible probes (with their requisite stiffeners) without damaging adjacent tissue -
a challenging task that may be difficult to achieve in practice.
Within the last few years, introduction of modern microelectronics technology, particularly
through the development of the (rigid) Neuropixel probes, has resulted in significant
Moreaux et al.
Page 6
Neuron
. Author manuscript; available in PMC 2021 October 14.
Author Manuscript
Author Manuscript
Author Manuscript
Author Manuscript
upscaling of multiplexed electrophysiology (
Jun et al., 2017
;
Steinmetz et al., 2019
). Recent
efforts by the Neuralink team (
Musk, 2019
) are also upscaling flexible probes that are
packaged with application-specific integrated circuits (ASICs) for neural recording. The
approaches used to connect the probes to the electronics today limit the number of
achievable channels, but this will continually improve as packaging technologies evolve.
This scaling, however, has largely been devoted to studying multiple or otherwise spatially
disparate brain regions, rather than to realize dense brain circuit coverage within a specific
region. For this latter and very important challenge, practical and fundamental limits
constrain the multiplicity of multi-site silicon probes for Ephys. These limitations result
from the inherent locality of electrical recording, which makes only a few neurons accessible
to any specific electrode within an array. For this reason, if
dense
recording (full volumetric
coverage of active neurons) appears to be very challenging to achieve by conventional
extracellular Ephys. Accordingly, it is natural to ask if alternative technologies exist that
might provide dense coverage of brain circuits with far fewer implanted elements. The key
here is to vastly increase the recording multiplicity obtainable from
each
recording element
within an implanted multiplexed array, thereby minimizing volumetric displacement, while
massively upscaling recording density and volumetric coverage. We will focus on this
overarching challenge throughout of the remainder of this article.
Beyond conventional Ephys: Free-space functional imaging
Functional imaging is an alternative approach to Ephys that enables both recording and
localization of neurons in anatomical space. It also provides the very important added
capability of specificity of cell type. However, as brain functions are not
optical
processes,
neuronal activity must be
transduced
from its intrinsic domain (Figure 1b) into the optical
domain; this is achieved by activity
reporters
. These reporters can be specific
macromolecules or nanoparticles that comprise two moieties: one, the
sensor,
reacts to a
targeted physical domain of local activity - be it electrical potential, molecular recognition,
or mechanical forces; the other, the
chromophore,
provides optical functionality - it
fluoresces at a particular wavelength when excited by an incoming photon within a specific
band of wavelengths. In response to local neural activity from one of the aforementioned
multiphysical measurement domains (Figure 1), the sensor in turn, modulates the optical
susceptibility of the chromophore. These bipartite reporters are then continuously
interrogated optically to determine their instantaneous state of activity. Hence, local
neuronal activity is directly reflected by changes in the chromophore’s optical susceptibility.
Functional imaging at depth
Currently, functional imaging of neuronal activity in the rodent cortex is widely achieved
using free-space multi-photon laser-scanning microscopy (
Denk et al., 1990
;
Lecoq et al.,
2019
), with brain tissue that is labeled by molecular reporters (
Grienberger and Konnerth,
2012
). This combination readily provides cellular resolution of neural activity. Among such
reporters are exogenous synthetic molecules, providing no cellular specificity; or genetically
encoded proteins that, as described below, provide cellular specificity through restriction of
their expression to specific cell types.
Moreaux et al.
Page 7
Neuron
. Author manuscript; available in PMC 2021 October 14.
Author Manuscript
Author Manuscript
Author Manuscript
Author Manuscript
Free-space, two- and three- photon, laser-scanning microscopy (
Denk et al., 1990
; Tolias)
together with fluorescent calcium reporters (
Chen et al., 2013
) enable functional imaging
with sub-cellular resolution. However, at increasing depths within the brain, scattering and
absorption ultimately preclude delivery of ballistic
(i.e.,
unscattered) light with sufficient
intensity and focus to achieve multi-photon excitation of specific reporter-labeled neurons.
The ultimate depth of delivery is limited by the optical attenuation length,
L
A
∼ 1/
L
sc
−1
+
L
ab
−1
, where
L
sc
and
L
ab
are the wavelength-dependent scattering and
absorption lengths, respectively (
Wang et al., 2018
). Further compounding this is the
challenge of extracting the information- bearing visible-wavelength fluorescent photons
emitted by reporters. They are even more strongly scattered;
L
a
at green wavelengths is
≲ 100
μ
m
(Figure 6a). This results from Mie scattering (
Bohren and Huffman, 2004
), which
is predominant in this regime; it greatly diminishes the photon yield available to free-space
optics placed
outside
the brain. Accordingly, even state-of-the-art
three-photon
functional
imaging provides cellular resolution solely at depths less than ~1.7mm (
Ouzounov et al.,
2017
). Despite significant effort and investment, the growing consensus among experts is
that it’s unlikely this range can be extended much further. All current methods for free-space
imaging are therefore applicable solely at rather shallow (< 2mm) tissue depths (e.g.,
cortex), or to transparent organisms (e.g., zebrafish larvae).
Microendoscopy and microfiber-based imaging
The complications outlined above have motivated the development of
microendoscopy.
This
method employs an optical fiber implanted in targeted brain regions, sometimes with a
miniature lens or prism, to achieve,
e.g.,
calcium functional imaging at the fiber’s distal end
via one-photon (1p) or two-photon (2p) fluorescence excitation. (
Liberti et al., 2017
;
Zong et
al., 2017
) Although such direct implantation resolves the issue of light delivery and recovery
from deep within the brain, the approach has several limitations:
i)
imaging is achieved only
within the optical plane near the endoscope tip,
ii)
tissue along the path of implanted
cannula/fiber (typically 0.3–2mm dia.) is destroyed and, hence,
iii)
current implementations
of this approach do not permit functional imaging along extended vertical regions (e.g.,
multiple cortical layers) simultaneously. Accordingly, the approach is generally feasible only
for acute measurements around the fiber/lens tip, using direct CMOS imaging (Inscopix) or
confocal laser microendoscopy (Mauna Kea). Finally,
iv)
it seems unlikely that this method
can be scaled up to achieve the dense volumetric coverage of neural activity that we consider
here.
Another strategy, closely related to the integrated neurophotonics paradigm described herein,
involves use of implantable tapered, optical fibers patterned to enable multi-point
illumination. These are coupled to an external laser source to enable passive, multi-point
brain illumination at depth (
Pisanello et al., 2017
;
Pisanello et al., 2014
). By adjusting the
incident light angle at the input fiber facet, various optical modes within the fiber can be
addressed. A number of these modes are preferentially coupled to patterned optical windows
along the length of the tapered fiber, so modal selection enables, in turn, site-selective light
delivery. Similarly-patterned tapered optical fibers can also can permit passive local
collection of light from a small number of sites along the fiber, when coupled to an external
photodetector (
Pisano et al., 2019
). Here again, microscale structuring of the fiber permits
Moreaux et al.
Page 8
Neuron
. Author manuscript; available in PMC 2021 October 14.
Author Manuscript
Author Manuscript
Author Manuscript
Author Manuscript
modal selectivity of the collected light. This approach enables depth profiling by fluorimetry
for structures in close proximity to the fiber. This approach has recently been combined with
a multi-electrode array (
Sileo et al., 2018
), and more recently a wireless system (
Emara et
al., 2018
).
These advancements provide considerable advantages over conventional optical fibers,
including the smaller, tapered form factor that is more amenable to tissue insertion, as well
as finer-scale light delivery,
i.e.,
finely patterned multi-point sources of light. A drawback,
particularly for photodetection, is the fact that the brain interrogation area achievable by this
approach is restricted to cells in close proximity to the fiber. Also, the passive light
collection used in this approach,
i.e
., guiding photons from the light source within the brain
via an optical fiber to an external photomultiplier, differs from the active, highly multiplexed
in situ
light collection employed for photonic neural probes in our work, described below.
The results of Pisannelo,
et al.
represent an early validation of the new paradigm of
implantable imaging systems with microscale dimensions. However, as tapered/window-
bearing fiber fabrication is carried out one-by-one, this technology is not directly compatible
with foundry-based mass production. Thus, it appears difficult to adapt this approach to
permit the massive upscaling of multiplicity required to enable dense, volumetric-scale
interrogation of brain activity over extended brain regions that we envisage here.
Status quo: Large-scale, volumetric functional imaging
One prominent recent example of large-scale volumetric functional imaging
in vivo
is the
multi-institution IARPA MICrONS project. In this effort, functional calcium imaging of all
excitatory neurons expressing GCaMP6 within a ~1 mm
3
volume spanning the mouse
primary visual cortex and higher visual areas was obtained using a wide field “mesoscope”
(
Sofroniew et al., 2016
;
Walker et al., 2019
). For each mouse studied, multiple scans tiling
the visual areas and cortical layers were obtained; these comprised many imaging planes
acquired at a spatial resolution of 0.4μm/pixel and a temporal resolution of 6.3 Hz. Over
5000 neurons were imaged simultaneously, thereby enabling functional characterization of
approximately 70,000 cells within each mouse. Once functionally imaged, the mice were
sectioned and imaged by electron microscopy (EM) with nanometer-scale resolution at the
Allen Institute of Brain Science. For one mouse, sectioning and imaging the complete
~1mm
3
volume spanning these regions by EM was performed. The EM data were
subsequently provided to Princeton University where the separate sectional images were
aligned, segmented (to identify every soma, axon, dendrite, and synapse of the ~100,000
cells within this tissue sample), and reconstructed in 3D. This combination of dense
functional imaging and EM-based anatomical reconstructions within the same tissue volume
is an important first step towards understanding relationships between the structure and
function of neural circuits. Such efforts will ultimately permit deciphering circuit-level
mechanisms that connect brain computations with behavior.
Molecular reporters and optogenetic actuators
To date, the most widely employed approach for functional imaging involves intracellular
Ca
2+
sensing (
Charpak et al., 2001
;
Yuste and Katz, 1991
). The temporal evolution of
Moreaux et al.
Page 9
Neuron
. Author manuscript; available in PMC 2021 October 14.
Author Manuscript
Author Manuscript
Author Manuscript
Author Manuscript
intracellular calcium concentration provides a robust proxy for direct electrophysiological
measurements (
Charpak et al., 2001
;
Ding et al., 2017
;
Moreaux and Laurent, 2007
), but the
approach has important limitations (
Moreaux and Laurent, 2007
). These molecular reporters
operate by sensing the calcium influx to the cell following an action potential; the resulting
change in concentration modulates binding of the calcium to the reporter’s Ca-sensing
moiety and, thereby, this induces a change to the optical susceptibility of its chemically
attached chromophore. The resulting stereotypical fluorescent transient that results is
interrogated optically to provide a “report” on calcium influx after the neuron fires
(
Moreaux and Laurent, 2007
,
2008
). This has become widely adopted (
Grienberger and
Konnerth, 2012
;
Yuste, 2010
) owing to the development of the excellent new class of fast
optogenetically-based calcium reporters exemplified by GCaMP6 (
Chen et al., 2013
).
Optical
stimulation
of neural activity requires optogenetic actuators (
Miesenböck, 2011
).
The most successful and widely deployed of these are derived from the Opsin family
(
Shichida and Matsuyama, 2009
). Opsins are light-sensitive ion channels and pumps that
transport specific ions across membranes in response to optical stimuli (
Zhang et al., 2007
)
(
Boyden, 2011
). Embedded within the cell membrane, these actuators can induce or block
action potentials when irradiated with light within a specific wavelength band.
Genes that encode for these molecular reporters and optogenetic actuators are introduced
into neurons, either by gene delivery methods using viral vectors or through genetic
engineering to create transgenic animal lines (
Luo et al., 2018
). This process enables
neurons to express exogenous GCaMP and Opsins. Selective expression of optogenetic
actuators and molecular reporters restricted to specific cell types is achieved through the use
of promoters that provide specificity of cell type. This offers enhanced selectivity and
enables controlled or sparse expression of optical reporters within brain tissue.
Advanced calcium reporters
One- and two-photon imaging of neurons expressing genetically encoded fluorescent
reporters of calcium concentration has become widely adopted in neuroscience. This is
because calcium signals are robust, with intracellular calcium rising dramatically in
concentration in many neuron types as a byproduct of firing action potentials, and because
bright, high dynamic range, fast, genetically encoded reporters can be created by fusing
fluorescent proteins to well-known calcium-binding protein motifs. The widely-employed
GCaMP family of calcium reporters (
Chen et al., 2013
;
Dana et al., 2019
;
Tian et al., 2009
),
for example, are based upon green fluorescent protein (GFP). Modern versions of these
reporters, such as GCaMP6 and GCaMP7, can reliably report both well-separated action
potentials and enable estimation of the frequencies of fast series of action potentials.
However, expressing a calcium reporter throughout a neuron results in fluorescent light not
only being generated from cell bodies, which many investigators want to focus on, but also
from neuronal processes,
i.e.,
from any axons and dendrites that are also illuminated. As
many axons and dendrites pass within an optical diffraction limit of a cell body, this can lead
to
neuropil contamination
during dense brain circuitry imaging. In this situation, optical
signals from nearby axons and dendrites contribute artifactual spikes to a cell body of
interest, and thus lead to artifactual activity correlations between neurons. Much effort has
Moreaux et al.
Page 10
Neuron
. Author manuscript; available in PMC 2021 October 14.
Author Manuscript
Author Manuscript
Author Manuscript
Author Manuscript
been invested in focusing or
patterning
light to improve the collection of calcium signals
from specific cells. However, complementary recent efforts have pursued what one might
call
molecular focusing
- that is, fusing calcium reporters to protein motifs that will localize
them preferentially within the cell body. Two recent efforts on such
somatic localization
fuse
GCaMP calcium reporters to different proteins, including a coiled-coil peptide set that
restricts GCaMP to the cell body (Figure 4a, lower panel) (
Shemesh et al., 2020
), and a
protein that tethers GCaMP to ribosomes, which also restricts GCaMP to the soma (
Chen et
al., 2020
). In both cases, neuropil contamination is significantly suppressed due to reduced
axonal and dendritic GCaMP, while somatic GCaMP brightness remains high and its
kinetics remain fast. This improvement serves to suppress incorrectly attributed spikes and
artifactual correlations between neurons. These benefits are observed in the mouse brain as
well as in other species, and they enhance functional imaging with both one- and two-photon
instrumentation (including microendoscopy). Thus, by lessening the reliance on optics for
selecting information to be obtained from specific cells, these “molecular focusing”
strategies help to clean up signals in a way that is complementary to optical focusing
methods. This can facilitate use of simpler, more scalable optical systems than currently
employed for imaging of neural dynamics
in vivo.
This “molecular focusing” approach has
also been applied to fluorescent reporters of transmembrane potential (Fig. 4), as discussed
in the following section.
Voltage reporters
As with optical calcium reporters, genetic approaches have also been used to design optical
voltage reporters, often referred to as genetically encoded fluorescent voltage indicators
(GEVIs). GEVIs are capable of reporting
subthreshold
voltage dynamics, which are not
resolvable using extracellular electrodes placed adjacent to individual neurons (
Herreras,
2016
). Further, as with all of the genetically encoded optical reporters described here,
genetic restriction - both in terms of cell type and subcellular location - can greatly facilitate
data extraction and analysis (i.e.
optical de-mixing
and back-end computational analysis, as
described below). To this end, in 2018 Daigle
et al.
reported the first somatically targeted
GEVI, which was achieved by fusing ASAP2s to a cytosolic segment of a potassium
voltage-gated channel (Kv2.1); this provided subcellular localization (Figure 4c) (
Daigle et
al., 2018
). This approach was subsequently employed with ASAP3 and Archon, to create
ASAP3-Kv (
Villette et al., 2019
) and SomArchon (
Piatkevich et al., 2019
) (Fig. 4b),
respectively.
Additional parameters essential for optical de-mixing of GEVI signals include the reporter’s
fluorescence amplitude
in response to single action potentials, and its
molar brightness
. The
largest relative fluorescence responses to individual action potentials are currently provided
by the red-excitable SomArchon (
ΔF/F
= 20–50% per AP) (
Piatkevich et al., 2019
) and the
blue-excitable ASAP3-Kv (Δ
F/F
= 10–30% per AP) (
Villette et al., 2019
). Due to a
difference in molar brightness between the two (0.076 mM
−1
cm
−1
for SomArchon versus
15 mM
−1
cm
−1
for ASAP3-Kv), illumination of ASAP3-Kv at ~25 mW/mm
2
at the focal
plane achieves the same SNR as illumination of SomArchon at 400 mW/mm
2
(
Villette et al.,
2019
). ASAP3-Kv and SomArchon thus provide two GEVI options at different wavelengths,
with ASAP3-Kv requiring less power delivery to tissue.
Moreaux et al.
Page 11
Neuron
. Author manuscript; available in PMC 2021 October 14.
Author Manuscript
Author Manuscript
Author Manuscript
Author Manuscript
One key drawback of GEVIs is the relatively fast sampling rate required. Actually, this
drawback originates from the relative slowness of present-day instrumentation; fast-
responding GEVIs such as Ace-NeonGreen or ASAP3 require sampling rates of > 500 Hz
dynamics to optimally track their fast temporal response (
Gong et al., 2015
;
Villette et al.,
2019
). Used in conjunction with 2p excitation, the instrumentation’s sampling rates restrict
the number of points that can be sampled over a spatially-limited plane given the rather slow,
serial nature of 2p microscopy (
Villette et al., 2019
;
Wu et al., 2020
). By contrast,
genetically encoded calcium reporters respond to calcium transients triggered by action
potentials that last for >100ms. For these slower responding reporters, frame rates of 15 to
30 Hz are sufficient to track reporter dynamics.
Integrated neurophotonic probes offer the possibility to record GEVIs “at speed,” given their
intrinsically fast temporal dynamics, which (as described below) are sufficiently fast to
follow even the ns-scale temporal decay of the chromophores (
Choi et al., 2019
). In
addition, since light delivery by neurophotonic probes is much more strategically delivered
within the illuminated tissue volume, background epifluorescence will be reduced compared
to free-space 1p methodologies.
Neurochemical reporters
The development of genetically encoded, intensiometric and ratiometric fluorescence-based
neurochemical reporters make it possible to perform direct, long-term, and chemically
specific functional imaging of neurotransmitters and neuromodulation dynamics. A range of
targets are now accessible, including dopamine, norepinephrine, serotonin, melatonin and
opioid peptides (
Leopold et al., 2019
;
Oe et al., 2020
;
Patriarchi et al., 2018
;
Sun et al.,
2018
). Generally speaking, there are two main types,
i.e.
design approaches, used to develop
neurochemical reporters: G protein-coupled receptor (GPCR)-based reporters and
periplasmic binding protein (PBP)-based reporters (
Andreoni et al., 2019
;
Leopold et al.,
2019
;
Ravotto et al., 2020
). A recent example of the former, the GPCR-based dLight1
chemical reporter family (
Patriarchi et al., 2018
) couples conformational changes of inert
human dopamine receptors to changes in the fluorescence intensity of circularly permuted
GFP (cpGFP) (Fig. 4d, left panel). This provides a direct read-out of dopamine kinetics with
broadly tunable affinity and dynamic range, relatively rapid kinetics (10 ms on and 100 ms
off), and fast temporal resolution that matches the performance of electrochemical methods
for detecting monoamines - while also providing subcellular resolution and molecular
specificity. Additionally, the presence of the dopamine receptor transmembrane domain
provides cell membrane targeting (Figure 4d, right panel).
In addition to GPCRs, bacterial PBPs have also been adapted for use as scaffolds for
engineering small molecule reporters. Their use exploits the conformational change that
occurs following binding of small molecules by its ligand binding domain (
Leopold et al.,
2019
), sometimes referred to as a “Venus Flytrap” domain (by analogy), which shares
similarity with binding domains of many eukaryotic chemical receptors (including GPCRs)
(
Acher and Bertrand, 2005
;
Felder et al., 1999
;
O’Hara et al., 1993
). As with GPCR-based
reporters, when coupled to a chromophore, this conformational change can be translated into
a change in the chromophore’s optical susceptibility and, thus, its fluorescence (
Leopold et
Moreaux et al.
Page 12
Neuron
. Author manuscript; available in PMC 2021 October 14.
Author Manuscript
Author Manuscript
Author Manuscript
Author Manuscript
al., 2019
). PBP-based neurochemical probes, which include probes for glutamate
(iGluSnFR), GABA (iGABASnFR), acetylcholine (iAchSnFR) and serotonin (iSeroSnFR),
are brighter, have larger dynamic range, lower affinity, and faster kinetics compared to
GPCR sensors. These attributes may mitigate problems such as buffering of native chemistry
within the cytosol and interference with endogenous receptors (
Marvin et al., 2018
;
Marvin
et al., 2019
).
The optical cross-sections of these chemical reporters are now similar to those of GCaMP
reporters; hence, future advancements in subcellular targeting (for example, dendritic
targeting of dopamine sensors) offers further potential for new insights. As the toolbox of
genetically encoded optical reporters continues to grow, combining them with other
reporters for simultaneous measurement of different functional read-outs becomes a way to
dissect multiphysical information processing within brain circuits occurring in diverse
physical domains (Figure 1). With the development of red-shifted dLight1 variants and the
new classes of calcium and voltage reporters, use of a multicolor approach offers the
possibility of investigating, in real-time, the simultaneous correlated activity of
neurotransmitters, neuromodulators and spiking.
Chromophore excitation
Currently, serial scanning methods based on two-photon microscopy are widely employed to
excite the chromophores within optical reporters. This involves simultaneous absorption of
two photons to induce nonlinear excitation of the reporter. Subsequently, decay of the
excited chromophore back to its ground state results the emission of a fluorescent photon in
the visible spectrum. Near-infrared excitation wavelengths are typically used for biological
microscopy given their longer attenuation lengths (Figure 6a). However, as the 2p optical
cross-section is very small, extremely high photon density is required to induce 2p
absorption. Accordingly, to achieve requisite intensities, 2p excitation requires use of a
single, tightly
spatially focused
beam of pulsed light that is also
temporally focused
into
femtosecond-scale pulses. To achieve volumetric sampling under these conditions, a serial
point-scanning methodology becomes necessary. The typical two-photon interrogation
voxel, generally of order ~0.5 × 0.5 × 2μm
3
, is thus scanned in 3D, one point at a time, to
spatially map the activity-dependent fluorescence of reporters within an ensemble of
neurons one-by-one. Today’s 2p-microscopes employing state- of-the-art acousto-optic
deflectors (AODs) enable provide down to ~1μs point-access time and optical spike
detection in multiple neighboring cells (~20) (
Villette et al., 2019
). This approach currently
permits routine mapping of ~400 neurons in a 3D volume of 200 × 200 × 100μm
3
with the
requisite signal-to-noise-ratio (SNR) to track spiking activity via the resulting modulation of
somatic calcium signals (
Cotton et al., 2013
;
Grewe et al., 2010
;
Katona et al., 2012
).
The aforementioned approach has two important limitations that greatly complicate attempts
to scale it up to enable functional imaging of large neuronal ensembles over extended brain
regions: (i)
serial optical interrogation, and (ii) SNR degradation with depth. We discuss
each in turn below.
Moreaux et al.
Page 13
Neuron
. Author manuscript; available in PMC 2021 October 14.
Author Manuscript
Author Manuscript
Author Manuscript
Author Manuscript
Multiplexing limits of free-space optical interrogation
Serial point-scanning optical techniques can provide sub-cellular resolution, but they have
the significant disadvantage that the total number of scanned voxels is limited by scanner
speed. This is exacerbated by the photometric requirement to dwell at each voxel long
enough to collect enough photons to attain requisite SNR. Simultaneous use of multiple
excitation beams has enabled multiplexing by in-plane parallelization of two-photon
microscopy with regular wide-field detection. In this implementation, each beam is encoded
with specific binary amplitude modulation to identify the location where fluorescence is
generated (
Ducros et al., 2013
).
Depth multiplexing
using four pulsed laser beams, has also
been developed; this has been successfully applied to mapping cortical activity in four
optical planes at four different depths (
Beaulieu et al., 2020
;
Cheng et al., 2011
). While these
approaches enable significant multiplexing, in practice only a relatively small number of
beams can be implemented. The maximum benefit obtained is ultimately determined by the
number of beams multiplexed, the laser repetition rate, the reporter fluorescence decay time,
and the total optical power that can be absorbed by brain tissue without undue perturbation
to neuronal activity (or the tissue itself).
Signal-to-noise ratio limits to the depth of imaging
Scattering and absorption limit the ability to deliver ballistic
(i.e.,
unscattered) light with
sufficient intensity to achieve two-photon excitation deep within the brain. Ultimately,
absorption limits the depth of delivery; in the near infrared (NIR) the maximum attenuation
length is
La
~500μm (Figure 6a). To overcome this significant limitation, several approaches
have been explored. In one, the instantaneous laser power is increased to enable deeper two-
photon excitation, while the pulse repetition rate is reduced to minimize the average power
delivered to the tissue. This approach enables recording neuronal activity in populations of
L5 neuronal soma up to ~800μm deep (
Mittmann et al., 2011
). However, collecting the
visible- wavelength fluorescence photons from brain tissue becomes especially problematic
at increasing depths. For these photons, scattering becomes the predominant limitation - as
described below.
Extending this approach to achieve even deeper functional imaging becomes increasingly
challenging; among issues are the generation of out-of-focus fluorescence (even with
moderate spatial confinement along the beam), and the onset of nonlinear photodamage in
neural tissue. A recent alternative approach involves using longer excitation wavelengths in
the near infrared (NIR) around 1.6mm. This becomes possible if three-photon absorption
processes are harnessed, and remarkable progress in this area has been achieved (
Horton et
al., 2013
). Here, the principal complication is the
significantly
smaller 3p cross-sections for
existing reporters; this imposes limits on the utility of this methodology, as much higher
illumination intensities are required. An alternative approach employs adaptive optical
corrections to rectify wavefront aberrations arising from spatially inhomogeneous optical
scattering and absorption in brain tissue (
Yaqoob et al., 2008
) (
Girkin et al., 2009
). This can
effectively restore optical resolution in the 2p modality and can significantly improve deep-
imaging capability. The approach is contingent upon measuring and employing the precise
aberration matrix
for a large volume of heterogeneous tissue. This is a challenging prospect;
it requires complex multi-point measurements and subsequent computations. As brain
Moreaux et al.
Page 14
Neuron
. Author manuscript; available in PMC 2021 October 14.
Author Manuscript
Author Manuscript
Author Manuscript
Author Manuscript
topology is dynamical, the correction matrix remains effective only for the interval over
which tissue is, in effect, stationary. Future work will elucidate the realm of applicability of
this approach.
Structured Illunimination
To separate fluorescent signal sources, laser-scanning microscopy localizes the illumination
in space and time. We term this
precisely structured illumination.
Other fluorescence
microscopy modalities structure their collection fields, sorting emitted photons by the place
of their origin. For example, in wide-field microscopy signals are localized by focusing
emitted light to form a spatial image. When neither the illumination fields nor collection
fields can be structured with sufficient spatial and temporal resolution, fluorescent signals
become mixed. In this case, computational approaches can be employed to separate or infer
the underlying signals. Several recording and imaging modalities (including some
fluorescence imaging modalities) use computational approaches to infer the image or optical
signals from insufficiently structured illumination and collection.
In diffuse optical tomography (DOT), multiple spatially separated light emitters illuminate
the tissue (usually one at a time), and the resulting scattered light is subsequently collected
by optical detectors at many different spatial locations. In this modality, spatial resolution is
increased by increasing the number of sources and detectors. This technique has yielded
resolution of ~ 4mm for imaging blood oxygenation level dependent (BOLD) effects
through the skull (
Dehghani et al., 2008
;
Wheelock et al., 2019
). Time-dependent diffusive
optical tomography (TD-DOT) approaches, in which the emitters are pulsed and the time-of-
flight (ToF) of the photons is determined at each of the detectors, improves the imaging
capabilities by allowing better separation of scattering effects from those resulting from
absorption or fluorescence (
Azizi et al., 2009
;
Painchaud et al., 1999
;
Puszka et al., 2013
).
Light-sheet fluorescence microscopy
(LSFM), also known as selective-plane illumination microscopy, is one example of a
compromise in which some degree of structured illumination is maintained while achieving
higher frame rates than are possible in a point-based laser-scanning system, and use of
focused imaging to separate collected light. LSFM is a rapid, wide-field, volumetric imaging
technique that enables volumetric imaging with optical sectioning (
Chen et al., 2014
;
Hillman et al., 2019
). In LSFM, a thin sheet of excitation light is generated, either by
cylindrically focusing a beam or by digitally scanning a Gaussian or Bessel beam (
Keller et
al., 2008
;
Mertz, 2011
;
Power and Huisken, 2017
). The sheet is scanned linearly across the
sample as fluorescence images are sequentially collected perpendicular to the illumination
plane. Stacking these sheet images then forms the desired volumetric image (
Huisken et al.,
2004
;
Keller and Ahrens, 2015
). A complication of conventional LSFM is its requirement
for two orthogonal objective lenses that must be specifically positioned spatially. This has
largely constrained application of the technique to quasi-transparent organisms (e.g., larval
zebrafish,
C. elegans, Drosophila
embryos) (
Ahrens et al., 2013
;
Chen et al., 2018
),
chemically cleared mammalian brains (
Keller and Ahrens, 2015
), and brain slices
(
Haslehurst et al., 2018
). Recently, swept confocally-aligned planar excitation (SCAPE)
microscopy, an LSFM method requiring only a single objective, has been developed to
Moreaux et al.
Page 15
Neuron
. Author manuscript; available in PMC 2021 October 14.
Author Manuscript
Author Manuscript
Author Manuscript
Author Manuscript
circumvent these constraints. With SCAPE,
in vivo
calcium neural imaging has been
demonstrated in mice (
Bouchard et al., 2015
;
Voleti et al., 2019
).
Acousto-optical techniques
Another approach to improving imaging depth is scattering tissue is to employ ultrasound-
modulated optical tomography (UOT), also known and acousto-optic imaging (
Resnik et al.,
2012
;
Wang and Zhao, 1997
). This has been used successfully to imaging absorption (
Wang
et al., 1995
) and scattering (
Kothapalli et al., 2007
) at depth in tissue. Light is passed
through an ultrasound beam. When it does so, it undergoes a frequency shift by multiples of
the ultrasound frequency; by detecting the frequency-shifted light, resolution can be
determined by the properties of the ultrasound (
Wang, 2001
). This tagging can be done at
the source of emission. This approach is very similar to photoacoustic tomography (PAT)
(
Xu and Wang, 2006
). In this case, acousto-optical effects are used to produce ultrasound
upon optical illumination and this ultrasound is used for imaging. While ultrasound is much
more penetrative than light, these techniques are still limited in depth by absorption of
ultrasound energy, which increases with carrier frequencies. Higher frequencies are required
to improve wavelength-determined resolutions. While imaging is possible at depths up to
several centimeters, cellular resolution is not attainable at depths beyond ~3mm (
Liu and Li,
2020
).
Implantable microscopes
Several realizations of head-mounted microscopes for 1p and 2p calcium imaging in mice
have proven the feasibility of fluorescence microscopy in compact form factors (
Corder et
al., 2019
;
de Groot et al., 2020
;
Ghosh et al., 2011
;
Jacob et al., 2018
;
Shuman et al., 2020
;
Skocek et al., 2018
;
Zong et al., 2017
). To achieve imaging at depth, these instruments
require implantation of the requisite GRIN lenses for these microscopes (typically 0.3 – 2
mm dia.), which results in rather significant displacement of brain tissue. Extending these
devices to support LSFM requires generating light sheets parallel to the surface of the brain
at arbitrary depths while ensuring tissue damage is minimal after implantation of the
requisite elements. In (
Ye et al., 2016
), a light sheet was generated by a microchip using a
nanophotonic grating coupler, a rather thick (>100 μm) glass spacer element, and a metallic
slit lens. In another demonstration, a sizable millimeter-scale prism coupled to a GRIN lens
for light sheet delivery was implanted alongside a second imaging GRIN lens (
Engelbrecht
et al., 2010
). These examples remain limited by the significant tissue displacement of their
implantable elements and their capability to generate only a single, static, light sheet.
Emission-related limitations
As mentioned, scattering in neural tissue drastically suppresses the yield of fluorescence
photons that can be collected outside the brain via free-space optics. Fluorescent photons
originating deep from within the brain are multiply scattered as they pass through tissue to
its periphery. To efficiently capture them, free-space collection optics with large angular
acceptance,
i.e.
large field of view and low magnification have been employed (
Oheim et al.,
2001
). These can be rather costly and physically immense. To date, they have provided
rather modest benefits.
Moreaux et al.
Page 16
Neuron
. Author manuscript; available in PMC 2021 October 14.
Author Manuscript
Author Manuscript
Author Manuscript
Author Manuscript
The integrated neurophotonics paradigm
To surmount the limitations of free-space and endoscopic functional imaging described
above, we have conceived of a new paradigm that we term
integrated neurophotonics.
It can
provide the basis to enable fast and dense volumetric mapping of brain activity. It leverages
recent advances in integrated silicon nanophotonics, nanoelectronics, and optogenetics, to
enable massively multiplexed functional imaging arbitrarily deep within the brain.
Employed together with optogenetic actuators and molecular reporters, these photonic
neural probe arrays - realized by integrating all elements of a lens-less imaging system onto
ultranarrow implantable silicon shanks - can enable dense interrogation of brain activity with
minimal tissue displacement (Figures 5 & 7). As such, it contains elements of many of the
approaches described above including leverage implantable CMOS electronics (as in scaled
Ephys approaches), structured illumination (as in LFSM or scanning microscopy), and
computation approaches that attempt to exact as much information as possible from
scattered photon (as in diffuse optical tomography). Unlike DOT, however, this brings the
emitters and detectors closer to the fluorescent sources, producing finer structure in local
illumination and collection fields to resolve local signals even in a turbid medium such as
brain tissue.
Photonic-probe-based interrogation is unique in its ability to circumvent the scattering
limitations of brain tissue. This is achieved by distributing the imaging components -
specifically, geometrically-ordered arrays
(architectures)
of microscale photon emitters (
E-
pixels
) and detectors
(D-pixels)
- within the brain, separated only by distances of order a few
times
L
A
(Figure 5e). The depth limitation for photonic probe functional imaging then solely
arises from the readily-engineerable probe length and the actual depth of their implantation.
This new paradigm requires the integration of a lens-less imaging system onto narrow
implantable shanks. It also requires the creation of new back-end hardware - to control the
instrumentation and stream the vast amounts of data it will produce. And, it requires new
and efficient computational algorithms that transform the immense cache of raw data the
system produces into a succession of time-sequenced, cell-specific functional recordings of
neuronal activity. Together, these elements form a complete and ultrafast lens-less functional
imaging system with an implantable front-end having microscale dimensions. Probe
modules comprise an ordered array of shanks that can be readily implanted at arbitrary
depths anywhere within the brain - either as an individual module, or as coherent module
arrays to permit dense and extended volumetric coverage.
A fundamental concept: Source localization in highly scattering brain
tissue
A pervasive mindset is that achieving cellular level resolution of sources in diffuse media,
even at modest distances, is impossible because (as the thought goes) light scattering in
neural tissue quickly and completely randomizes the direction of emerging photons. These
challenges, for example, lead to the limitations of conventional DOT. For integrated
neurophotonic systems, this problem is surmounted by positioning microscale emitters and
detectors within the brain tissue separated by distances of order a few times
L
A
. This is
Moreaux et al.
Page 17
Neuron
. Author manuscript; available in PMC 2021 October 14.
Author Manuscript
Author Manuscript
Author Manuscript
Author Manuscript