of 26
Nature | Vol 629 | 23 May 2024 |
861
Article
Temporal multiplexing of perception and
memory codes in IT cortex
Liang She
1
, Marcus K. Benna
2,3
, Yuelin
Shi
1
, Stefano
Fusi
2
& Doris
Y.
Tsao
1,4,5
A central assumption of neuroscience is that long-term memories are represented by
the same brain areas that encode sensory stimuli
1
. Neurons in inferotemporal (IT)
cortex represent the sensory percept of visual objects using a distributed axis code
2
4
.
Whether and how the same IT neural population represents the long-term memory of
visual objects remains unclear. Here we examined how familiar faces are encoded in
the IT anterior medial face patch (AM), perirhinal face patch (PR) and temporal pole
face patch (TP). In AM and PR we observed that the encoding axis for familiar faces is
rotated relative to that for unfamiliar faces at long latency; in TP this memory-related
rotation was much weaker. Contrary to previous claims, the relative response
magnitude to familiar versus unfamiliar faces was not a stable indicator of familiarity
in any patch
5
11
. The mechanism underlying the memory-related axis change is likely
intrinsic to IT cortex, because inactivation of PR did not affect axis change dynamics
in AM. Overall, our results suggest that memories of familiar faces are represented in
AM and perirhinal cortex by a distinct long-latency code, explaining how the same cell
population can encode both the percept and memory of faces.
Our experience of the world is profoundly shaped by memory. Whether
we are shopping for a list of items at the grocery store or talking to
friends at a social gathering, our actions depend critically on remember
-
ing a large number of visual objects. Multiple studies have explored the
molecular
12
,
13
and cellular
14
,
15
basis for memory, but the network-level
code remains elusive. How is a familiar song, place or face encoded by
the activity of neurons?
Recent work on the sensory code for visual object identity in the
inferotemporal (IT) cortex suggests that objects are encoded as points
in a continuous, low-dimensional object space, with single IT neurons
linearly projecting objects onto specific preferred axes
2
4
(Fig.
1a
, left).
These axes are defined by weightings of a small set of independent
parameters spanning the object space. This coding scheme (also
referred to as linear mixed selectivity
16
,
17
, and related to disentangled
representations in machine learning
18
) is efficient, allowing a huge
number of different objects to be represented by a small number of
neurons. Indeed, the axis code carried by macaque face patches allows
detailed reconstruction of random realistic faces using activity from
only a few hundred neurons
3
.
Here we set out to leverage recent insight into the detailed sensory
code for facial identity in IT cortex
3
to explore the population code for
face memories. A long-standing assumption of neuroscience is that
long-term memories are stored by the same cortical populations that
encode sensory stimuli
1
. This suggests that the same neurons that carry
a continuous, axis-based, object-coding scheme should also support
tagging of a discrete set of remembered objects as familiar. However,
schemes for representing discrete familiar items often invoke attrac-
tors
19
,
20
that would lead to breakdowns in continuous representation
(Fig.
1a
, right). This raises a key question: does familiarity alter the IT
axis code for facial identity? We surmised that discovering the answer
might uncover the neural code for face memory.
Previous studies have generally found decreased and sparsened
responses to familiar stimuli in IT and perirhinal cortex and have
proposed that this decrease, or ‘repetition suppression’, is the neural
correlate of object memory
5
11
. However, these studies were not tar
-
geted to specific subregions of IT cortex known to play a causal role in
discrimination of the visual object class being studied
21
and where the
visual feature code is precisely understood
3
. Here, to study the neural
mechanism that represents long-term object memories, we targeted
three regions: anterior medial face patch (AM), the most anterior face
patch in IT cortex
22
, and PR and TP, two recently reported face patches
in the perirhinal cortex and anterior temporal pole, respectively
23
,
24
.
These three regions lie at the apex of the macaque face patch system,
an anatomically connected network of regions in the temporal lobe
dedicated to face processing
22
,
25
29
. AM harbours a strong signal for
invariant facial identity
3
,
22
, perirhinal cortex is known to play a critical
role in visual memory
30
33
and TP has recently been suggested to provide
a privileged pathway for rapid recognition of familiar individuals
24
. We
thus hypothesized that a representation of face memory should occur
in AM, PR and/or TP.
Our recordings showed that, in all three patches, familiar faces
were distinguished from unfamiliar faces. First, in all three patches,
familiar faces were represented in a subspace distinct from unfamiliar
faces. Second, in all three patches the relative response magnitude
to familiar faces differed significantly from that to unfamiliar faces;
however, the sign of this difference was not stable and depended
strongly on the relative frequency of presentation of familiar and unfa
-
miliar faces (that is, temporal context). Third, and most strikingly,
https://doi.org/10.1038/s41586-024-07349-5
Received: 19 March 2021
Accepted: 25 March 2024
Published online: 15 May 2024
Open access
Check for updates
1
Division of Biology and Biological Engineering, Caltech, Pasadena, CA, USA.
2
Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York City, NY, USA.
3
Neurobiology
Section, Division of Biological Sciences, University of California, San Diego, San Diego, CA, USA.
4
Howard Hughes Medical Institute, University of California, Berkeley, CA, USA.
5
Present address:
Department of Neuroscience, University of California, Berkeley, CA, USA.
e-mail:
liangshe@caltech.edu
;
dortsao@berkeley.edu