of 10
PNAS
2023 Vol. 120 No. 32 e2221122120
https://doi.org/10.1073/pnas.2221122120
1 of 10
Functional modules for visual scene segmentation in macaque
visual cortex
Janis K. Hesse
a,b,1
and Doris Y. Tsao
a,b,1
Contributed by Doris Y. Tsao; received December 12, 2022; accepted June 26, 2023; reviewed by Guy A. Orban and Daniel Tso
RESEARCH ARTICLE
|
NEUROSCIENCE
Segmentation, the computation of object boundaries, is one of the most important
steps in intermediate visual processing. Previous studies have reported cells across
visual cortex that are modulated by segmentation features, but the functional role of
these cells remains unclear. First, it is unclear whether these cells encode segmentation
consistently since most studies used only a limited variety of stimulus types. Second,
it is unclear whether these cells are organized into specialized modules or instead
randomly scattered across the visual cortex: the former would lend credence to a
functional role for putative segmentation cells. Here, we used fMRI-
guided electro-
physiology to systematically characterize the consistency and spatial organization of
segmentation-
encoding cells across the visual cortex. Using fMRI, we identified a set
of patches in V2, V3, V3A, V4, and V4A that were more active for stimuli containing
figures compared to ground, regardless of whether figures were defined by texture,
motion, luminance, or disparity. We targeted these patches for single-
unit recordings
and found that cells inside segmentation patches were tuned to both figure-
ground
and borders more consistently across types of stimuli than cells in the visual cortex
outside the patches. Remarkably, we found clusters of cells inside segmentation patches
that showed the same border-
ownership preference across all stimulus types. Finally,
using a population decoding approach, we found that segmentation could be decoded
with higher accuracy from segmentation patches than from either color-
selective or
control regions. Overall, our results suggest that segmentation signals are preferentially
encoded in spatially discrete patches.
segmentation | border ownership | modularity | figure-
ground
Segmentation is believed to be a crucial step in visual processing that allows a visual system
to distinguish which regions of a visual scene belong to the background, which regions
belong to different objects, and where the borders are that constrain the regions of these
objects. Previous studies suggest that the brain uses a diversity of cues, including lumi-
nance, texture, disparity, and motion, to segregate objects from the background (1–4).
Several major candidates for segmentation signals have been identified in visual cortex,
including figure-
ground encoding cells that respond more strongly to a figure in the
receptive field than to background (3–5), cells that encode curvature (6, 7), cells that
encode kinetic boundaries for motion-
defined figures (8–10), and so-
called
border-
ownership cells that respond more strongly if a border in the receptive field
belongs to an object on one specific side of it (2). To date, these various types of
segmentation-
encoding cells have been recorded in random locations of macaque visual
areas V1, V2, V3, and V4. However, a recent study suggests that at least border-
ownership
cells may be clustered within V4 (11). The general functional organization of
segmentation-
encoding cells is thus a question of great interest. Moreover, since in the
studies above only a limited number of stimulus types were presented, it is unclear
whether cells exist that encode figure-
ground or border ownership consistently, invariant
to whether the segmentation is defined by luminance, texture, motion, disparity, or
higher-
level cues. Indeed, one study found that putative border-
ownership cells recorded
from random locations in V2 and V3 do not encode border ownership consistently when
presented with a larger battery of artificial and natural stimuli (12). However, this result
leaves open the possibility that consistent cells can be found when recording from
appropriate functional modules.
Computational studies have shown that neural network models for object recognition
can benefit from separating representations of object appearance and object contours
(13–16). These findings hint at the possibility that visual cortex may exploit this modular
organization, with some parts of visual cortex specialized for extracting texture, and other
parts specialized for computing the segmentation of a visual scene. A vast amount of
literature has reported modularity in retinotopic cortex for different simple features, such
as color in thin stripes and motion/disparity in thick stripes (17). In contrast, here, we
Significance
Segmentation of a visual scene is
one of the most important steps
in visual processing as it allows
one to distinguish between
regions of the scene that belong
to different visual objects.
Previous studies reported cells
encoding different aspects of
segmentation in random locations
of visual areas, but a systematic
characterization of segmentation
tuning properties across visual
areas is missing. Here, we
combined fMRI and
electrophysiology to identify
“segmentation patches” that
preferentially encode
segmentation features. Our
findings suggest visual cortex may
be organized into modules not
only for specific features such as
orientation but also for a specific
computation, segmentation. This
advances our understanding of
visual cortical organization and
makes the question of how the
brain computes segmentation
more tractable.
Author affiliations:
a
Department of Molecular and Cell
Biology, University of California, Berkeley, CA 94720;
and
b
Helen Wills Neuroscience Institute, University of
California, Berkeley, CA 94720
Author contributions: J.K.H. and D.Y.T. designed
research; J.K.H. performed research; J.K.H. analyzed
data; and J.K.H. and D.Y.T. wrote the paper.
Reviewers: G.A.O., Universita degli Studi di Parma; and D.T.,
State University of New York Upstate Medical University.
The authors declare no competing interest.
Copyright © 2023 the Author(s). Published by PNAS.
This open access article is distributed under
Creative
Commons Attribution-
NonCommercial-
NoDerivatives
License 4.0 (CC BY-
NC- ND)
.
1
To whom correspondence may be addressed. Email:
janishesse@googlemail.com or tsao.doris@gmail.com.
This article contains supporting information online at
https://www.pnas.org/lookup/suppl/doi:10.1073/pnas.
2221122120/-
/DCSupplemental
.
Published July 31, 2023.
OPEN ACCESS
Downloaded from https://www.pnas.org by George Porter on September 11, 2023 from IP address 131.215.225.154.