of 12
RESEARCH ARTICLE
NEUROSCIENCE
ENGINEERING
Internal feedback in the cortical perception–action loop enables
fast and accurate behavior
JingShuangLi
a,1
ID
,AnishA.Sarma
a,b,1
,TerrenceJ.Sejnowski
c,d,2
ID
,andJohnC.Doyle
a,2
ID
ContributedbyTerrenceSejnowski;receivedJanuary9,2023;acceptedJuly18,2023;reviewedbyMarkSchnitzerandDomitillaDelVecchio
Animals move smoothly and reliably in unpredictable environments. Models of sensori-
motor control, drawing on control theory, have assumed that sensory information from
the environment leads to actions, which then act back on the environment, creating a
single, unidirectional perception–action loop. However, the sensorimotor loop contains
internal delays in sensory and motor pathways, which can lead to unstable control. We
show here that these delays can be compensated by internal feedback signals that flow
backward, from motor toward sensory areas. This internal feedback is ubiquitous in
neural sensorimotor systems, and we show how internal feedback compensates internal
delays. This is accomplished by filtering out self-generated and other predictable
changes so that unpredicted, actionable information can be rapidly transmitted toward
action by the fastest components, effectively compressing the sensory input to more
efficiently use feedforward pathways: Tracts of fast, giant neurons necessarily convey
less accurate signals than tracts with many smaller neurons, but they are crucial for fast
and accurate behavior. We use a mathematically tractable control model to show that
internal feedback has an indispensable role in achieving state estimation, localization
of function (how different parts of the cortex control different parts of the body),
and attention, all of which are crucial for effective sensorimotor control. This control
model can explain anatomical, physiological, and behavioral observations, including
motor signals in the visual cortex, heterogeneous kinetics of sensory receptors, and the
presence of giant cells in the cortex of humans as well as internal feedback patterns and
unexplained heterogeneity in neural systems.
internal feedback
|
speed–accuracy trade-off
|
optimal control
|
sensorimotor control
Feedback control is an essential strategy for both engineered and biological systems
to achieve reliable movements in unpredictable environments (1). Optimal and robust
control theory, which provides a general mathematical foundation to study feedback
systems, has been used successfully to explain behavioral observations by modeling the
sensorimotor system as a single control loop, also called the perception–action cycle
or perception–action loop (2–4). In these models, the sensorimotor system senses the
environment, communicates signals from sensors to the brain, computes actions, and then
acts on the environment, feeding back to the sensors and forming a single unidirectional
loop as shown in Fig. 1.
Consider the canonical model of localized function in the primate visuomotor cortical
pathway, depicted in Fig. 2: A visual signal is encoded on the retina, then travels to
the lateral geniculate nucleus (LGN) of the thalamus, and on to the primary visual
cortex (V1), progressing through successive transformations until it reaches the primary
motor cortex (M1), the spinal cord, and ultimately the muscles. Although intuitive,
this feedforward model neglects a well-known and ubiquitous feature of sensorimotor
processing: internal feedback, which is the main focus of this paper (5).
The perception–action control model does not have a direct role for internal feedback
connections. Internal feedback includes all signals that do not flow from sensing toward
action. We can divide internal feedback into two broad categories: counterdirectional
to feedforward projections and lateral interactions within or between areas at the same
processing level. Counterdirectional internal feedback is in the opposite direction of the
single-loop model (for instance, from V2 to V1); these signals flow from action toward
sensing. Lateral internal feedback consists of recurrent connections that are used for
exchanging information within a cortical area (for instance, within V2), or between areas
at the same level in parallel streams (such as between areas MT and IT in the dynamical
and object recognition streams, respectively). This distinction emphasizes the importance
of where signals are spatially located in cortical hierarchies (Fig. 2).
The single-loop model offers a set of tools from control theory and a conceptual
framework that allows subsystems to be treated as successive transformations that can be
Significance
Internal feedback
projections—signals flowing from
motor areas or late sensory
processing regions back to early
sensory processing regions such
as primary visual and auditory
areas—are ubiquitous in the
sensorimotor nervous system
and are as or more numerous
than feedforward projections.
However, the function of internal
feedback is poorly understood,
particularly in the context of task
performance. We leverage
control theory and simple models
to demonstrate that internal
feedback facilitates good task
performance when there are
communication limitations such
as internal time delays and
speed–accuracy trade-offs, which
motivate compensatory feedback
signals to counter self-generated
and predictable movements.
Control theory explains why
motor-related signals are found
throughout the sensory cortex
and why the motor cortex is
dominated by internal dynamics.
Author contributions: J.S.L., A.A.S., T.J.S., and J.C.D.
designed research; performed research; and wrote the
paper.
Reviewers: M.S., Stanford University; and D.D.V.,
Massachusetts Institute of Technology.
The authors declare no competing interest.
Copyright
©
2023 the Author(s). Published by PNAS.
This article is distributed under Creative Commons
Attribution-NonCommercial-NoDerivatives License 4.0
(CC BY-NC-ND).
1
J.S.L. and A.A.S. contributed equally to this work.
2
To whom correspondence may be addressed. Email:
doyle@caltech.edu or terry@salk.edu.
Published September 22, 2023.
PNAS
2023 Vol. 120 No. 39 e2300445120
https://doi.org/10.1073/pnas.2300445120
1 of 12
Downloaded from https://www.pnas.org by George Porter on November 2, 2023 from IP address 131.215.249.205.
Fig. 1.
Single-loop model of sensorimotor control. The organism receives
information from the external environment via sensors, communicates this
information through the body, computes actions, and then acts on the envi-
ronment; this forms the external feedback loop, or single loop model (black).
Internal signals that flow opposite to the direction of the external feedback
loop are classified as internal feedback (pink). Thus, the internal feedback is
counterdirectional. Internal feedback also includes lateral interactions within
an area or between areas at the same processing stage (not shown).
studied in isolation. However, these subsystems are not isolated.
With internal feedback, each subsystem has access to bottom
up, top down, and lateral information. The eye is itself a site of
computation and control: as the eye moves and senses different
parts of the visual scene, lateral interactions within the retina
control spatial and temporal filter properties that can adapt and
identify important features under a wide range of illumination
and scene dynamics (6, 7). Retinal ganglion cells project to relay
neurons in the LGN, which then project to the primary visual
cortex, V1, but a much greater number of feedback neurons
project from V1 to LGN (8–10) (Fig. 2).
Projections from motor and later sensory areas in the cortex to
early visual areas have a wide range of morphology, myelination,
and synaptic kinetics (8, 11, 12). Given the position of M1
in the final common pathway, one might expect activity in
M1 to be driven by current visual stimuli or current move-
ments, but instead, autonomous internal dynamics dominate
the data (13). Counterintuitively, signals related to movements
of the whole body are found in areas typically associated with
particular parts of the body, such as the hand area, as well
as sensory areas such as the primary visual cortex (14–17).
Indeed, recent analysis of the correlation structure between
neurons during a visual discrimination task revealed a task-
related global mode in the correlations between cortical neurons
associated with the task response rather than the sensory stimulus,
strongly supporting the idea that
Top–Down
feedback is an
important element of sensory processing (18). These motor-
related signals in sensory pathways, which span subsystems and
tasks, are generated by internal feedback and are the focus of
this study.
Internal feedback has been studied in the context of sensory
predictive coding (19, 20) and has been invoked in other
modeling studies and theory frameworks (21–25). However,
these models focus on sensory or motor systems separately and do
not account for key constraints on neuronal communication in
both space and time to achieve sensorimotor tasks. Achieving fast
and accurate computation and communication across brain areas
is difficult, or even impossible, because communication may be
slow, limited in bandwidth and constrained to spatially localized
populations.
Here, we build on the foundations of recent work in distributed
control theory (26–30) and show that internal feedback is a
solution to achieving rapid and accurate control given the spatial
and temporal constraints on brain components and communica-
tion systems. We analyze an idealized class of control models
and prove mathematically that internal feedback is necessary
for achieving optimal performance in these idealized models.
Internal feedback serves at least three functions: state estimation,
localization of function, and focused attention, all of which are
crucial for effective sensorimotor control and survival. This
theory explains why there are differences in population responses
between M1 and V1, why different projections predominantly
activate AMPA or NMDA glutamate receptors, the functions
of giant pyramidal cells in visuomotor control, and both the
uses and limitations of localization of function in the cortex.
There is a general principle behind all of these physiological
properties.
Task Model and Performance.
We analyze expected values and
theoretical bounds on task performance for highly simplified
control loop models motivated by a well-studied and ethologically
relevant tracking task—reaching for a moving object. The goal
of the task is continuous pursuit, such as catching a fly ball in
baseball, rather than one-time contact between limb and object.
The complete tracking task requires identification of the object
in a cluttered visual scene, prediction of the object’s movement,
and generation and execution of bimanual limb movement. We
make many simplifying assumptions that allow us to study
internal feedback in an accessible way using familiar linear
dynamical systems. Models of greater detail and complexity are
discussed briefly.
Single-loop feedback control.
Consider the task of tracking a
moving object with the endpoint of a limb on a plane. The
variable to be controlled is the tracking error—the distance
between the hand and the object. We start by assuming that
the system controlling the limb can perfectly sense the position
of the limb and object at every instant, which will be relaxed in
/
/
/
/
Fig. 2.
A partial, simplified schematic of sensorimotor control. We focus
on key cortical and subcortical areas and communications between them.
Black and green arrows indicate communications that traverse from sensing
toward actuation; green arrows are particularly fast pathways, which enable
the tracking of moving objects in our model. Pink arrows indicate internal
feedback signals, which traverse from actuation toward sensing. Solid lines in-
dicate direct neuronal projections, while broken lines include both direct and
indirect connections. SC, spinal cord; Th, thalamus; V1, primary visual cortex;
M1+, primary motor cortex and additional motor areas; V2/3, secondary and
tertiary visual cortex; IT, inferotemporal cortex; MT, mediotemporal cortex
(V5). Only a subset of the internal feedback pathways are shown (e.g., not
included are internal feedback signals from M1+ to V2 and IT).
2 of 12
https://doi.org/10.1073/pnas.2300445120
pnas.org
Downloaded from https://www.pnas.org by George Porter on November 2, 2023 from IP address 131.215.249.205.
later models. The cost is defined as the squared Euclidean norm
of the tracking error over time, normalized by the total amount
of time, with a smaller cost indicating better tracking.
Let
x
,
u
, and
w
represent the tracking error, the control action
on the limb, and the action of the object, respectively. We will
refer to
x
as the state of the system. Let
A
be a matrix that
represents the intrinsic dynamics of
x
, including features such as
the movement of the object or mechanical coupling between two
dimensions of limb movement. The time-evolving dynamics of
the tracking error follows from a linear equation of motion:
x
(
t
+
1
) =
Ax
(
t
) +
u
(
t
) +
w
(
t
)
.
[1]
In general, the difficulty of a task will depend on properties of
A
such as its eigenvalues and the strength of coupling between
states. For example, if the spectral radius of
A
is less than 1, this
corresponds to a task in which tracking error
x
decreases with no
limb action, an easy task.
The actions
u
provide feedback control on the tracking error,
computed by an arbitrary function
K
that has access to all past
and present tracking errors
x
(
1
:
t
)
, as follows:
u
(
t
) =
K
(
x
(
1
:
t
))
.
[2]
The optimal solution to this problem is the linear quadratic
regulator (LQR) and the optimal controller is
K
(
x
(
1
:
t
)) =
Ax
(
t
)
if the action of the object
w
behaves as white noise
(1). This controller is compatible with the single-loop model of
sensorimotor control, as there is no internal feedback, and the
addition of internal feedback does not provide any additional
performance advantage.
Controllers without internal feedback are optimal for a
large but special class of problems, including standard state
feedback and full control problems from control theory. Though
mathematically elegant, these controllers make assumptions that
are impractical when applied to biological systems. In subsequent
sections, we relax some of the assumptions implicit in this single-
loop model and show that small deviations from assumptions
relevant to biological systems introduce the need for internal
feedback.
Any of the controllers in subsequent sections can be imple-
mented in a variety of ways, although whether or not a particular
controller needs internal feedback is generic across all possible
implementations. We choose particular nonunique controller
implementations with internal feedback for which the optimal
solution is relatively transparent and easy to interpret.
State Estimation Requires Counterdirectional Internal Feed-
back.
Internal feedback facilitates implicit estimation in the presence
of sensor delays.
Simple modifications to the control problem
described above lead to an optimal controller
K
whose implemen-
tation requires internal feedback. One such modification is the
introduction of sensor delays, which are ubiquitous in biological
systems (for example, the neuronal conduction time from the
eye to the motor cortex is on the order of tens to hundreds of
milliseconds). Sensor delays can be modeled by introducing a
virtual internal state
x
s
, which represents the adjusted tracking
error from the previous time step (29). This formulation allows us
to pose the delayed-sensor tracking problem as a standard control
problem which can be optimally solved by a linear quadratic
regulator (LQR):
̃
A
=
[
A
0
I
0
]
, C
= [
0
I
]
[
x
(
t
+
1
)
x
s
(
t
+
1
)
]
=
̃
A
[
x
(
t
)
x
s
(
t
)
]
+
u
(
t
) +
[
w
(
t
)
0
]
[3]
u
(
t
) =
KC
[
x
(
t
)
x
s
(
t
)
]
,
where the virtual internal state
x
s
contains delayed information
about the tracking error. Controller
K
can be partitioned into
two block-matrices, (
K
=
[
K
>
1
K
>
2
]
>
). The resulting system is
shown in Fig. 3. Here, the controller does not directly “perceive”
the tracking error
x
and only has access to the virtual internal state
x
s
. However, the controller can freely take actions that affect both
the tracking error and the virtual state. The action on the virtual
state, as shown in Fig. 3, is an example of counterdirectional
internal feedback with gain
K
2
.
For the delayed sensing problem, the optimal controller has
a simple analytical form:
K
2
=
A
is the internal feedback and
K
1
=
A
2
. If no internal feedback is allowed (i.e., we enforce
K
2
=
0), then the optimal controller is
K
1
=
A
2
/
4. We
compare the performance of these two controllers in Fig. 4 and
see that the controller with internal feedback far outperforms the
controller without internal feedback. We also note that as the task
becomes more difficult (spectral radius of
A
>
2), the controller
without internal feedback is unable to stabilize the closed-loop
system and tracking breaks down.
For a controller with sensory delays, internal feedback is
required for optimal performance. This also applies to controllers
with actuator delays (29). In both cases, internal feedback adjusts
delayed signals to compensate for actions taken and information
received during the delay; in other words, internal feedback
implicitly compensates for the delays.
The linear quadratic problem that we consider here could be
straightforwardly tested in a laboratory setting. A simple real-
world task described by linear dynamics is the action of a stable
limb, resting on a surface or manipulandum, tracking an object
over a line or plane. This can be modeled by neutrally stable
double-integrator dynamics (
A
=
[
I I
0
I
]
,
=
1) with single
time-step delays corresponding to internal loop delays on the
order of 100 ms. More complex models of tasks corresponding
to more realistic scenarios, such as movement against gravity or
adversarial disturbances, would tend to increase the penalty of
control without internal feedback.
+
2
1
Fig. 3.
Optimal control model for a system with sensor delays. Tracking
error
x
is sensed, and then communicated by the sensor with some delay to
the
K
1
block, which computes the appropriate actuation. Counterdirectional
internal feedback (pink) conveys information from actuation back toward
sensing. Internal computation
K
2
adjusts the sensor signal to compensate
for actions taken by the system; this results in improved performance.
PNAS
2023 Vol. 120 No. 39 e2300445120
https://doi.org/10.1073/pnas.2300445120
3 of 12
Downloaded from https://www.pnas.org by George Porter on November 2, 2023 from IP address 131.215.249.205.
Fig. 4.
Internal feedback improves performance when there are internal
delays in sensing. The scalar problem of tracking a moving target over a line
was simulated, varying the task difficulty (
= spectral radius of
A
, representing
the dynamics). The “Ideal” controller contains no sensor delays. The “Internal
Feedback” controller contains sensor delays, and uses internal feedback to
compensate for the delays. The “No Internal Feedback” controller contains
sensor delays but uses no internal feedback. As
approaches 2, the task
becomes infeasible without internal feedback (broken line). Shaded areas
indicate SDs.
We next describe controllers with internal feedback that
include sensor delays, actuator delays, and imperfect sensing that
will motivate a general model for sensorimotor control in brains.
IntrinsicinternalfeedbackintheKalmanfilter.
We now consider
the case in which sensing is instantaneous, but imperfect and
noisy, and actuation is imperfect. Consider the following system:
x
(
t
+
1
) =
Ax
(
t
) +
Bu
(
t
) +
w
(
t
)
y
(
t
) =
Cx
(
t
) +
v
(
t
)
,
[4]
where
y
is the sensor input and
v
is the sensor noise, assumed to be
white noise. Matrix
B
represents the effect of action
u
on tracking
error
x
, and matrix
C
represents how sensor input
y
is related to
tracking error
x
. This is a standard formulation in control theory,
and the optimal controller makes use of controller gain
K
and
estimator gain
L
as follows:
ˆ
x
(
t
+
1
) =
A
ˆ
x
(
t
) +
Bu
(
t
) +
L
(
y
(
t
)
C
ˆ
x
(
t
))
u
(
t
) =
K
ˆ
x
(
t
)
,
[5]
where
ˆ
x
is an internal estimate of tracking error
x
. This optimal
controller uses the Kalman filter, which inherently contains
three counterdirectional internal feedback pathways irrespective
of delays being present. These pathways are represented by
the blue arrows in Fig. 5 and play a central role in state
estimation. The pathway through
A
estimates state evolution
in the absence of noise and actuation; the pathway through
B
accounts for controller action, and the pathway through
C
predicts incoming sensory signals based on the internal estimated
state.
The implementation shown in Fig. 5 is not unique. We now
briefly discuss a few equivalent implementations that use less
internal feedback and explain why they are less advantageous in
the sensorimotor context than the implementation in Fig. 5.
In one alternative implementation, we can remove the internal
feedback through
B
and replace
A
with
A
+
BK
. However,
this requires duplication of
K
; in the sensorimotor context,
which requires duplicating of motor structures within visual
structures. In another alternative implementation, we can remove
the internal feedback through
C
and replace
A
with
A
LC
. This
requires a duplication of
L
. Additionally, filtering out predictable
sensory input via
C
earlier (as is done in Fig. 5) can be preferable
to filtering it out later (as in our alternative implementation).
This is because the filtered information is typically much smaller
in bandwidth and requires less resources to communicate: The
earlier we perform this filtering, the less resources we require to
pass this information forward. If communications are subject
to a speed–accuracy trade-off (described below), then earlier
filtering allows us to pass sensory information forward with
less delay.
SourcesofinternalfeedbackarepreservedinaKalmanfilterwith
delays.
We now synthesize a model that combines features from
previous sections: sensor delays, actuator delays, and imperfect
sensing. The model can be constructed using virtual states as
follows:
[
x
(
t
+
1
)
x
a
(
t
+
1
)
x
s
(
t
+
1
)
]
=
[
A B
0
0
0
0
C
0
0
][
x
(
t
)
x
a
(
t
)
x
s
(
t
)
]
+
[
0 0
I
0
0
I
]
[
u
(
t
)
u
s
(
t
)
]
+
[
w
(
t
)
0
0
]
y
(
t
) =
x
s
(
t
)
,
[6]
where
x
a
and
x
s
are virtual internal states corresponding to delayed
actuator commands and delayed sensor signals, respectively, and
u
s
represents compensation on virtual internal states. We can use
standard control theory to obtain the optimal controller gain
K
and optimal estimator gain
L
. Due to the block-matrix structure
of the system matrices, the optimal gains have the following
structure:
K
= [
K
1
K
2
0
]
, and
L
=
[
L
>
1
0
L
>
2
]
>
(29).
The controller can be implemented as follows:
(
t
+
1
) =
Cx
(
t
)
C
ˆ
x
(
t
)
L
2
(
t
)
ˆ
x
(
t
+
1
) =
A
ˆ
x
(
t
) +
Bx
a
(
t
) +
L
1
(
t
)
u
(
t
) =
K
1
ˆ
x
(
t
) +
K
2
x
a
(
t
)
,
[7]
where
is the delayed difference between the estimated sensor
input and true sensor input, adjusted by the
L
2
(
t
)
term. The
resulting controller, shown in Fig. 6, contains two internal
feedback pathways related to delay: one pathway compensates for
sensor delays, and the other compensates for actuator delays. The
remaining internal feedback is inherent to the Kalman filter, as
described in the previous section and shown in Fig. 5. Overall, the
+
-
+
Fig. 5.
Internal feedback in a controller with instantaneous but imperfect
sensing and actuation.
A
,
B
, and
C
represent the state, actuation, and sensing
matrices of the physical plant;
K
represents the optimal controller, and
L
represents the optimal observer. The Time Shift block shifts
ˆ
x
(
t
+
1
)
to
ˆ
x
(
t
)
in
Eq. 5. The internal feedback pathways (blue) are inherent to the Kalman filter;
these use state, actuation, and sensing models to create an internal estimate
of the tracking error or state. All internal feedback depicted in this diagram
is counterdirectional and assumed to have no delay and infinite bandwidth.
4 of 12
https://doi.org/10.1073/pnas.2300445120
pnas.org
Downloaded from https://www.pnas.org by George Porter on November 2, 2023 from IP address 131.215.249.205.
+
+
-
+
1
1
2
2
Fig. 6.
Internal feedback in a controller with sensor and actuator delays.
A
,
B
, and
C
represent the state, actuation, and sensing matrices of the
physical plant;
K
1
,K
2
,L
1
,andL
2
are submatrices of the optimal controller and
observer gains. The internal feedback pathways (pink) through
L
2
and
K
2
compensate for sensor and actuator delays, respectively. Other internal
feedback pathways (blue) are inherent to the Kalman filter. All internal
feedback depicted in this diagram is counterdirectional. The yellow box
contains parts of the controller that roughly correspond to motor areas in
the cortex.
inclusion of sensor delays, actuator delays, and imperfect sensing
results in an optimal controller with several internal feedback
pathways, each of which serves a specific, interpretable purpose.
Localization of Cortical Function Requires Lateral Internal
Feedback.
Almost all muscles in the body are engaged in even
the simplest actions, such as reaching. Controlling a system with
many degrees of freedom is a difficult problem for motor control
even without delays. Localization of function is well established
in the motor cortex, with different body parts controlled by
different cortical areas; however, communication and computa-
tion between localized cortical areas typically have spatial and
temporal constraints, compared with signals within areas.
Consider two motor areas and partition tracking errors into
two sets
x
1
and
x
2
, representing two distinct but possibly
coupled subsystems (e.g., two distinct limbs that are mechanically
coupled) using the problem formulation described by Eq.
1
. The
overall tracking error is
x
=
[
x
>
1
x
>
2
]
>
. Correspondingly, we
partition actuators into two sets
u
1
and
u
2
that act on their
respective subsystems via local controllers:
u
=
[
u
>
1
u
>
2
]
>
.
Each local controller senses and controls one subsystem; i.e.,
local controller 1 senses
x
1
and computes
u
1
, and local controller
2 senses
x
2
and computes
u
2
. Local controllers may communicate
with another; however, due to localization constraints, the cross-
communication is delayed. Thus, local controller 1 cannot
directly access
x
2
without some delay and similarly for local
controller 2.
We observe that without the constraint of localized commu-
nication, the optimal controller for Eq.
1
is
u
=
Ax
. If
A
is
block-diagonal (i.e.,
x
1
and
x
2
are uncoupled), then this controller
obeys localized communication constraints—in fact, no cross-
communication (internal feedback) is required between the two
local controllers. However, if the two subsystems are coupled,
then this controller requires rapid, global communication, which
violates localized communication constraints.
To enforce localized communication, we reformulate the
problem by introducing virtual states
x
1
and
x
2
, which represent
delayed cross-communication between the two local controllers.
x
1
is information sent from local controller 1 to local controller 2,
with delay; and similarly for
x
2
. We also define
u
1
and
u
2
, which
model interconnections between virtual states and real tracking
errors. For simplicity, we assume unit delay. The reformulated
problem then becomes
̃
x
=
x
1
x
2
x
1
x
2
,
̃
u
=
u
1
u
2
u
1
u
2
̃
w
=
w
1
0
0
w
2
,
̃
A
=
A
11
0 0
A
12
0
0 0
0
0
0 0
0
A
21
0 0
A
22
, K
=
∗ 4 ∗
0
4 ∗ 4 ∗
∗ 4 ∗ 4
0
∗ 4 ∗
̃
x
(
t
+
1
) =
̃
A
̃
x
(
t
) +
̃
u
(
t
) +
̃
w
(
t
)
̃
u
=
K
̃
x
.
[8]
The zeros in the
Top Right
and
Bottom Left
corners of the
K
matrix preserve localized communication; they enforce that
the two local controllers cannot communicate instantaneously
to one another. Asterisks and triangles indicate free values:
Triangles represent sites of potential cross-communication, or
lateral internal feedback. When these free values are optimized to
achieve optimal performance with localized communication, the
resulting
K
matrix is
K
=
A
11
0
A
12
0
0
0
A
12
A
12
A
21
A
21
0
0
0
A
21
0
A
22
.
[9]
The resulting local controllers are shown in Fig. 7. Note that
the
A
12
term in the second row and the
A
21
term in the fourth
row of
K
correspond to lateral internal feedback. Here, these
internal feedback signals carry predicted values of the unsensed
tracking errors for each controller, after taking control action into
account; for instance, internal feedback from local controller 2 to
local controller 1 conveys the predicted value of
x
2
, after taking
control action from controller 2 into account.
+
+
21
12
12
11
21
Fig. 7.
Optimal localized control of two coupled subsystems. (
Top
) Overall
schematic. Each subsystem has its own corresponding local controller,
which senses and actuates only its assigned subsystem. Local controllers
communicate to each other via lateral internal feedback (pink), with some
delay. (
Bottom
) Circuitry of local controller 1. Local controller 2 has identical
circuitry, with different matrices;
A
12
instead of
A
21
,
A
22
instead of
A
11
, etc.
PNAS
2023 Vol. 120 No. 39 e2300445120
https://doi.org/10.1073/pnas.2300445120
5 of 12
Downloaded from https://www.pnas.org by George Porter on November 2, 2023 from IP address 131.215.249.205.
We can develop intuition for this implementation by following
an impulse
w
through time:
̃
x
(
1
) =
w
1
0
0
w
2
→ ̃
x
(
2
) =
A
12
w
2
A
12
w
2
A
21
w
1
A
21
w
1
→ ̃
x
(
3
) =
0
.
[10]
The performance of this controller can be compared to the
controller without internal feedback in Fig. 8. The best possible
linear controller for the controller without internal feedback
results in severe performance degradation. As task difficulty
increases, this controller is unable to stabilize the closed-loop
system and tracking becomes infeasible. With internal feedback,
task performance stays near the centralized optimal case where
local controllers can communicate freely without delay.
This analysis shows that when motor function is localized to
specialized parts of the motor cortex that control particular parts
of the body, cross-communication via internal feedback between
local controllers is essential. Local circuits in the two hemispheres
must also be coordinated —indeed, they are connected by a
massive corpus callosum that crosses the midline.
The cross-talk between local controllers is supported by the
presence of global signals from movements of the whole body to
the local controller in the motor cortex specialized to particular
parts of the body. In reality, all body movements are mechanically
coupled, something which the motor system can conceal through
effective localization and coordination using internal feedback.
Speed–Accuracy Trade-Offs Drive the Use of Layering and
Internal Feedback for Attention.
We have shown that state
estimation and localization of function require internal feedback
to correct for self-generated or predictable movements. We now
show how to efficiently track a moving object with the limitations
imposed by neural components and internal time delays using an
attention mechanism.
Up to this point, we have assumed that the controller can
directly sense the position of the object (perhaps with some
delay). In the real world, a scene can comprise many objects,
Fig. 8.
Localization of function within the motor-related cortex: Although
different parts of the cortex control different parts of the body, these parts of
the body are inherently mechanically coupled. As a result, internal feedback
is useful and in some cases necessary to maintain localization of function.
In simulations, we consider the problem of tracking a moving target over a
two-dimensional space, varying the task difficulty. The “Ideal” controller is
centralized (i.e., no delays between local controllers) and obtains the best
performance. The localized controller with internal feedback achieves similar
performance. The localized controller without internal feedback suffers from
substantially worse performance (higher cost). As task difficulty increases,
the task becomes infeasible without internal feedback (broken line). Shaded
areas indicate SDs.
which makes it more difficult for a sensorimotor system to
localize an object in the scene. However, a moving object, once
identified, can be more easily discriminated from a static visual
scene. This illustrates the distinction between scene-related tasks
(such as object identification) and error-related tasks (such as
object tracking), which in the visual cortex is accomplished by
the ventral and dorsal streams, respectively.
This distinction also mirrors the separation between bumps
and trails in the mountain-biking task studied in ref. 27, allowing
us to build on the control architecture in that task. The main
difference is that instead of separating into two control loops,
we use layering and internal feedback to supplement the control
actions of the main control loop.
We consider a one-dimensional problem (tracking on a line)
and use as the metric
x
(worst-case tracking error for
adversarial object action) rather than
x
2
(average-case tracking
error for random object action). Worst-case error is a more
realistic model of many ethological tasks, and optimal solutions
to worst-case control problems can have additional internal
feedback pathways compared to average-case; however, the worst-
case setting is less familiar in neuroscience models than the
average-case setting we have considered to this point (31). We
have some object whose position,
r
, is governed by the dynamics
r
(
t
+
1
) =
r
(
t
) +
w
r
(
t
) +
w
b
(
t
)
,
[11]
where
w
r
represents object movement, and
w
b
represents changes
in the background scene. Limb position
p
is governed by the
dynamics
p
(
t
+
1
) =
p
(
t
)
u
(
t
)
,
[12]
where
u
(
t
)
is some limb action. The tracking error
x
:=
r
p
then obeys the dynamics
x
(
t
+
1
) =
x
(
t
) +
w
r
(
t
) +
w
b
(
t
) +
u
(
t
)
,
[13]
where the task difficulty is implicitly equal to 1.
We assume that object movement and background changes are
bounded:
|
w
r
(
t
)
|≤
r
and
|
w
b
(
t
)
|≤
b
for all
t
. Additionally,
we assume that background changes are much slower than object
movement:
b

r
, i.e.,
r
+
b
r
.
[14]
Consider a movable sensor that senses some interval of size
on the continuous line. Information from the sensor must
be communicated to the controller via axon bundles, which
are subject to speed–accuracy trade-offs— that is, the higher
bandwidth a signal, the slower it can be sent. Thus, roughly
speaking, axonal communication can be low-bandwidth and fast,
or high-bandwidth and slow. We can formalize this as follows
for a volume of cortex axons with uniform radius, adapting from
ref. 27:
We first observe that delay
T
is inversely proportional to axon
radius
with proportionality constant
:
T
=
.
[15]
Firing rate per axon,
, is proportional to axon radius with
proportionality constant
:
=
훽휌
.
[16]
Cross-sectional area
s
is related to axon radius
and the number
of axons in the nerve
n
via:
s
=
n
휋휌
2
.
[17]
6 of 12
https://doi.org/10.1073/pnas.2300445120
pnas.org
Downloaded from https://www.pnas.org by George Porter on November 2, 2023 from IP address 131.215.249.205.