TRANSFORMATIONS OF NEURAL REPRESENTATIONS IN A SOCIAL BEHAVIOR NETWORK

TRANSFORMATIONS OF NEURAL REPRESENTATIONS IN A

SOCIAL BEHAVIOR NETWORK

Bin Yang

1,2

Tomomi Karigo

1,2,4

David J. Anderson

1,2,3

Division of Biology and Biological Engineering 140-80, TianQiao and Chrissy Chen Institute for

Neuroscience, California Institute of Technology, Pasadena CA 91125 USA

Howard Hughes Medical Institute, California Institute of Technology, 1200 E California Blvd,

Pasadena CA 91125 USA

SUMMARY

Mating and aggression are innate social behaviors that are controlled by subcortical circuits in

the extended amygdala and hypothalamus

–

. The bed nucleus of the stria terminalis (BNSTpr)

is a node that receives input encoding sex-specific olfactory cues from the medial amygdala

(MeApd)

, and which in turn projects to hypothalamic nuclei that control mating

–

(MPOA) and

aggression

–

(VMHvl), respectively

. Previous studies have demonstrated that male aromatase

BNSTpr (AB) neurons are required for mounting and attack, and may identify conspecific sex

according to their overall level of activity

. However neural representations in BNSTpr, their

function and their transformations in the hypothalamus have not been characterized. Here we

have performed calcium imaging

of male BNSTpr

Esr1

neurons during social behaviors. We

identify distinct populations of female- vs. male-tuned neurons in BNSTpr, with the former

outnumbering the latter by ~2:1, similar to MeApd and MPOA but opposite to VMHvl, where

male-tuned neurons predominate

. Chemogenetic silencing of BNSTpr

Esr1

neurons whilst

imaging MPOA

Esr1

or VMHvl

Esr1

neurons in behaving animals revealed, unexpectedly, that the

male-dominant sex-tuning bias in VMHvl was inverted to female-dominant, while a switch from

sniff- to mount-selective neurons during mating was attenuated in MPOA. Our data also indicate

that BNSTpr

Esr1

neurons are not essential for conspecific sex identification. Rather, they control

the transition from appetitive to consummatory phases of male social behaviors by shaping sex-

and behavior-specific neural representations in the hypothalamus.

Author for correspondence: Tel: (626) 395-6821, FAX: (626) 564-8243, wuwei@caltech.edu.

Author contributions

B.Y. performed experiments, analyzed data, and prepared figures. T.K performed retrograde tracing experiments. B.Y. and D.J.A.

designed the study and wrote the paper.

Present address: Kennedy Krieger Institute, Johns Hopkins School of Medicine, 707 N. Broadway, Baltimore, MD 21205

Competing interests

The authors declare no competing interests.

HHS Public Access

Author manuscript

Nature

. Author manuscript; available in PMC 2023 February 03.

Published in final edited form as:

Nature

. 2022 August ; 608(7924): 741–749. doi:10.1038/s41586-022-05057-6.

Author Manuscript

BNSTpr

Esr1

neurons control social behaviors independently of conspecific

sex identification

We first examined the behavioral effects of silencing Esr1

neurons in the principal

subdivision of BNST (BNSTpr

Esr1

neurons) in socially experienced males. Chemogenetic

silencing

significantly reduced mounting and increased sniffing toward females, and

reduced attack toward males, consistent with prior observations in inexperienced males

(Extended Data Fig. 1a–m). Next, we optogenetically silenced

BNSTpr

Esr1

neurons

during either the appetitive or consummatory phases of social interactions with males

or females (Fig. 1). Silencing BNSTpr

Esr1

cells during approaching or sniffing of males

strongly inhibited the transition to, and duration of, attack, but did not significantly alter

sniffing itself (Fig. 1e–g and Extended Data Fig. 1n–q). Silencing BNSTpr

Esr1

neurons

during approaching or sniffing of females significantly reduced the transition to (~2 fold)

and duration of (~2.5 fold) mounting (Fig. 1j–l and Extended Data Fig. 1t), while extending

the total time spent sniffing (Extended Data Fig. 1r, s). Interestingly, while optogenetic

silencing during attack interrupted this behavior (Fig. 1h, i), silencing during ongoing

mounting had no effect (Fig. 1m, n). Optogenetic silencing of BNSTpr

Esr1

terminals

–

the medial preoptic area (MPOA) or ventromedial hypothalamus, ventrolateral subdivision

(VMHvl) revealed that the sniff-to-mount transition toward females is primarily dependent

upon activity in the BNSTpr

→

MPOA projection (Extended Data Fig. 2l), while activity

in the BNSTpr

→

VMHvl projection is necessary for attack (Extended Data Fig. 2i). These

data confirm that BNSTpr

Esr1

activity is required to gate the transition from appetitive to

consummatory male social behaviors towards both sexes

, and show that this gating

occurs via its projections to MPOA and VMHvl, respectively. They additionally demonstrate

a requirement for BNSTpr

Esr1

activity during ongoing attack (but not mounting)

A previous study concluded that BNSTpr

neurons are required for intruder sex

recognition, based on male vs. female cue-preference tests

. Males emit USVs in

response to female but not male urine

. We performed bilateral optogenetic silencing

of BNSTpr

Esr1

neurons using GtACR2 and tested both cue preference and USVs. Consistent

with earlier data

, we observed a reduced preference for female over male cues (Extended

Data Fig. 1r, w). Surprisingly, however, there was no loss of USVs towards female urine

or intact females (Extended Data Fig. 1s, x). Thus, while silencing BNSTpr

Esr1

neurons

reduced the males’ preference for female cues, it did not eliminate their ability to recognize

and distinguish female vs. male cues.

The male BNSTpr

Esr1

population represents conspecific sex via a cell

identity code

Previous bulk calcium measurements of BNSTpr

neurons suggested that intruder sex

is encoded by overall activity, such that strong activity indicates a female, while weaker

activity indicates a male

. To reveal how sex and social behaviors are represented in

BNSTpr at the single cell level, we imaged BNSTpr

Esr1

neurons (a subset partially

overlapping with, but larger than, AB neurons

) expressing jGCaMP7f

using a

microendoscope (Inscopix, Inc.; Fig. 2a). By utilizing a prism-coupled gradient refractive

Yang et al.

Page 2

Nature

. Author manuscript; available in PMC 2023 February 03.

Author Manuscript

index (GRIN) lens, we were able to image the dorso-ventral extent of this elongated

nucleus in behaving mice, yielding almost three-fold more units per session compared to

conventional cylindrical GRIN lenses (Fig. 2b, c; Extended Data Fig. 3a–h).

We first performed trials in which male, female, or toy mice suspended by their tails were

presented to male residents in their home cage (5 trials each, 10s per trial), permitting

sniffing but not mounting or attack

. We observed distinct BNSTpr

Esr1

populations

responding to either male or female “dangled” intruders, but not to toy mice (Fig. 2c,

f and Extended Data Fig. 4a). The average down-sampled bulk calcium peak response

(5 × 10 s trials) to female intruders was ~2 times that of male intruders (Fig. 2d),

explaining prior bulk calcium imaging results

. Analysis of single unit calcium responses

(extracted using CNMF-E

, see Methods) to intruders revealed that 27% of the units

were female preferring (determined by choice probability

(CP); see Methods) vs. 15%

that were male preferring (Fig. 2g, h and Extended Data Fig. 4e). Qualitatively similar

results were obtained by imaging of BNSTpr

neurons

, a functionally equivalent subset

of BNSTpr

Esr1

neurons

(Extended Data Fig. 4j–l). A temporal linear decoder

(Support

Vector Machine, SVM) trained on BNSTpr

Esr1

data could identify the sex of a dangled

intruder, in held-out test data, within 1 sec of intruder presentation (p<0.0001) compared

to shuffled control data (Fig. 2i), suggesting that olfactory cues are sufficient to generate

distinct cellular representations of intruder sex in BNSTpr. Male and female-preferring cells

were not topographically segregated but appeared intermingled (Fig. 2e).

We asked next whether BNSTpr

Esr1

population activity also represents distinct social

behaviors, by imaging under conditions where the resident male freely interacted with male

or female intruders (4,585 units from 15 mice). We constructed raster plots of population

activity during a 10s interval from each of 8 different conditions (baseline, sniff male/

female, interact with male/female, attack, mount, and intromission; see Methods) (Fig. 2j).

Overall, ~33% of imaged units were active (>2

) during female encounters, while ~25%

were activated during male encounters. Of these, ~45% were activated during appetitive

and/or consummatory behaviors toward the intruder (Fig. 2k, orange circle and Extended

Data Fig. 4b, c, d). Approximately 77% of the female-preferring (511 out of 660) and 70%

of the male-preferring (259 out of 377) units that exhibited a sex preference during dangling

trials retained this preference during subsequent unrestrained social interactions (Fig. 2l).

We computed the tuning of BNSTpr

Esr1

neurons for intruder sex or specific behaviors

using choice probability

, a binary comparison. Among all units that were active during

appetitive and/or consummatory behaviors, a larger percentage were male- or female-tuned

(20% male, 33% female), than were consummatory behavior-tuned (7% attack, 9% mount)

(Fig. 2m and Extended Data Fig. 4f). Principle Component (PC) analysis indicated that

the largest source of variance in neural activity was explained by intruder sex (Fig. 2n,

PC1). Linear regression (Methods) revealed that intruder sex accounted for over 30% of

the observed variance, while only 9% and 8% of the variance was explained by behavior

towards females and males, respectively (Fig. 2o and Extended Data Fig. 3i). Together, these

results suggest that a major source of variance in BNSTpr

Esr1

population activity is the sex

of the intruder (or an internal motive state highly correlated with intruder sex), and that sex

is represented by population coding, as in MPOA and VMHvl

Yang et al.

Page 3

Nature

. Author manuscript; available in PMC 2023 February 03.

Author Manuscript

Control of hypothalamic representations of conspecific sex by BNSTpr

Esr1

neurons

We next directly compared representations of intruder sex in MeApd, BNSTpr, VMHvl,

and MPOA Esr1

neurons, imaged under identical conditions (Fig. 3a–f; Extended Data

Fig. 3i). Like BNSTpr, both the MPOA and VMHvl Esr1

populations contained distinct

subsets of male- and female-preferring cells

. However, whereas among MPOA

Esr1

and

MeApd

Esr1

neurons female-preferring outnumbered male-preferring units, by about 2:1 (as

in BNSTpr

Esr1

), among VMHvl

Esr1

neurons the ratio was reversed (1:2; Fig. 3o, p, gray

bars; Extended Data Figs. 3i and 5p–s), consistent with earlier results

Since BNSTpr projects directly to MPOA and VMHvl

and encodes conspecific sex

identity

, we hypothesized that inhibiting BNSTpr

Esr1

neurons should eliminate sex-

specific representations in MPOA and VMHvl. To test this, we combined chemogenetic

silencing of BNSTpr

Esr1

neurons using hM4Di

, with microendoscopic imaging of

VMHvl

Esr1

or MPOA

Esr1

neurons expressing jGCaMP7f

(Fig. 3b,c and Extended Data

Fig. 5a–o), a method we call “ChemoScope”. Imaging during presentation of dangled

intruders was performed both in control (“pre-CNO”, i.e. saline-injected) and experimental

(CNO-injected) conditions in the same animals, to permit within-subject comparisons at the

single-cell level.

Surprisingly, silencing BNSTpr

Esr1

neurons during presentation of dangled male or female

intruders did not eliminate sex-specific representations in VMH and MPOA (Fig. 3e vs.

h; 3f vs. i). A small, but statistically significant, decrease in the separation of male vs.

female representations was observed in VMHvl, as indicated by the Pearson’s correlation

coefficient and Mahalanobis distance ratio

(Fig. 3j, k, VMHvl, opposite sex, gray vs.

maroon bars). However, the performance of linear decoders of intruder sex, trained on

data from MPOA

Esr1

or VMHvl

Esr1

neurons, was only slightly decreased in MPOA by

silencing BNSTpr

Esr1

neurons, and was still well above chance (computed using shuffled

data). Sex decoding in VMHvl was unaffected (Fig. 3 l–m). There was a slight increase in

the difference between average z-scored responses to females vs. males in MPOA, and a

larger increase in VMHvl (Fig. 3n), likely reflecting a decrease in the mean response to male

intruders in both nuclei, and an increase in the mean response to female intruders in VMHvl

(Extended Data Fig. 6q–s).

In contrast to these relatively subtle effects, silencing BNSTpr

Esr1

neurons unexpectedly

inverted the 2:1 ratio of male- to female-tuned units in VMHvl, to the female-dominant

ratio seen in MPOA (~2:1; Fig. 3p; VMH, gray vs. maroon bars). This reversal of sex

bias in VMHvl reflected both a decrease in male-preferring units, and an increase in female-

preferring units (Fig. 3o). In MPOA, by contrast, the ratio of female- to male-preferring

units was slightly increased by BNSTpr silencing (Fig. 3p, q and Extended Data Fig. 6s, t).

Together, these data indicate that the activity of BNSTpr

Esr1

neurons is not required for the

neural coding of intruder sex identity by MPOA

Esr1

and VMHvl

Esr1

neurons. Rather, it is

required to invert, in VMHvl, the female bias in population representations of intruder sex

seen in BNSTpr, MPOA and MeApd

, to a male bias.

Yang et al.

Page 4

Nature

. Author manuscript; available in PMC 2023 February 03.

Author Manuscript

The inversion of sex tuning bias in VMHvl caused by silencing BNSTpr

Esr1

neurons could

be explained by two mechanisms: conversion, in which a subset of initially male-tuned

VMHvl

Esr1

neurons switched to female-tuning; or selection, in which new female-tuned

units appeared and replaced silenced male-tuned units. To distinguish these possibilities,

we registered the spatial maps imaged during pre-CNO (saline-injected) and CNO sessions

within the same animals, and tracked single unit responses

(over 60% of all recorded

MPOA

Esr1

units and over 55% of all recorded VMHvl

Esr1

units) during male and female

dangling trials, before and after CNO treatment (Fig. 4 a, i). Consistent with the data from

all (i.e., non-tracked) units (Fig. 3p), we observed an approximate inversion of the ratio

of male- to female-preferring VMHvl

Esr1

neurons, from 2m:1f before, to 1m:1.7f after,

BNSTpr silencing (Fig. 4b, c). In contrast, the ratio of female- to male-preferring MPOA

Esr1

tracked units was slightly increased in CNO, from 1.5f:1m to 2f:1m (Fig. 4j, k; cf. Fig. 3p).

Single-unit tracking revealed the source of the additional female-preferring Esr1

units in

VMHvl following BNSTpr silencing (Fig. 4h). Almost a quarter of the female-preferring

units (24%) in CNO-treated animals derived from pre-CNO male-preferring units that

switched sex-preference (Fig. 4h, blue; Extended Data Fig. 7q, s, VMH, red bar). Twenty

percent of the female-preferring units after CNO derived from pre-CNO “co-active” (i.e.,

responsive to both sexes) units that lost male preference (Fig. 4h, yellow; Extended Data

Fig. 7l–n), or from initially non-active units that gained female preference (20%) (Fig. 4h,

grey; Extended Data Fig. 7g–i). Thus, the inversion of the male sex-tuning bias in VMHvl

following BNSTpr silencing was due mostly to the conversion of initially male-preferring,

or dual-responsive, units to female-preferring units (Fig. 4g).

In MPOA, following BNSTpr

Esr1

silencing a substantial proportion of initially male-

preferring, co-active or non-active units converted to female-preferring units (Fig. 4o).

However, this conversion was partially offset by the silencing of some initially female-

preferring units (Fig. 4o, Extended Data Fig. 7c,

right

; f, k, n). Consequently, the overall

proportion of female-preferring MPOA

Esr1

units did not change greatly after CNO (Fig.

4j, k, p). In contrast to VMHvl, therefore, where BNSTpr

Esr1

silencing converted initially

non-female-selective to female-selective units, in MPOA the slight increase in female-tuned

units (Fig. 3n, p) was primarily due to a substitution of new female-tuned units for others

that became inactive. There was no significant change in the percentage of tracked male-

or female-tuned units in either MPOA or VMHvl, in animals injected with saline on two

consecutive days (Extended Data Fig. 5p–s).

Control of hypothalamic representations of social behaviors by BNSTpr

Esr1

neurons

To investigate how silencing BNSTpr

Esr1

neurons affects hypothalamic activity during

mounting and attack, we tracked single-unit responses in MPOA

Esr1

or VMHvl

Esr1

neurons

before and after unilateral silencing of BNSTpr in freely behaving animals (Fig. 5a).

Because BNSTpr

→

hypothalamic projections are primarily ipsilateral

, by performing

silencing and imaging on one side of the brain while leaving the other unperturbed, this

design allowed normal social behavior even in CNO, due to redundant contralateral circuits.

Yang et al.

Page 5

Nature

. Author manuscript; available in PMC 2023 February 03.

Author Manuscript

Thus, any observed changes in neural activity on the silenced side cannot be due to a loss of

social behavior, which requires bi-lateral BNSTpr

Esr1

silencing (Fig. 1b and Extended Data

Fig. 1a).

In MPOA, female-directed sniffing vs. mounting were represented by largely distinct groups

of neurons (Fig. 5b, pre-CNO). CNO silencing of BNSTpr

Esr1

significantly decreased the

activity of mount-selective cells, with a concomitant increase in sniff-active cells during

mounting behavior (Fig. 5c–f). Thus, during BNSTpr

Esr1

silencing the cellular responses

during female-directed sniffing vs. mounting became more similar than in controls (Fig. 5b,

CNO). A corresponding decrease in the fraction of variance in neural activity explained by

female-directed behaviors was observed as well (Fig. 5g).

In contrast to MPOA, in VMHvl male-directed sniffing and attack are represented by largely

overlapping neuronal populations (Fig. 5h)

. Following BNSTpr

Esr1

silencing, both the

average activity of those neurons normally active during attack behavior, as well as well

as the percentage of attack-active neurons, were significantly reduced (Fig 5j–l). There was

also a decrease in the number of male-selective VMHvl

Esr1

neurons (Fig. 5i, CNO; see

also Fig. 4b, c). Together these data show that in MPOA, unilateral BNSTpr

Esr1

silencing

extended the activity of normally sniff-selective cells into the mounting phase, and decreased

the activity of mount-selective cells during mounting. In VMHvl it reduced the activity of

both mixed-selectivity and attack-selective cells during the attack, but not the sniff, phase.

Thus, silencing of BNSTpr

Esr1

neurons in freely behaving males during social interactions

caused distinct changes in neural representations of consummatory behavior in VMHvl

and MPOA. However the average response to males and females in both nuclei remained

unchanged (Fig. 3g

♀♂

Avg.).

Discussion

BNSTpr lies at the interface between the encoding of sex-specific olfactory cues in

the medial amygdala and of motive drive states in the hypothalamus that control the

consummatory phases of social behaviors. Previous studies suggested that it represents

and is essential for the identification of conspecific sex via an intensity code

, with its

requirement for consummatory behaviors presumably reflecting the former function. The

data presented here suggest an alternative view.

First, while we confirm that BNSTpr neural activity contains a representation of conspecific

sex identity, our cellular resolution calcium imaging reveals that female vs. male identity

is not encoded by a higher vs. lower level of activity in this structure, respectively, as

indicated by previous bulk calcium measurements

, but rather by a cell identity code:

distinct subpopulations of BNSTpr

Esr1

neurons are tuned to males vs. females. Because the

female-tuned neurons outnumber the male-tuned cells by ~2:1, a higher level of BNSTpr

activity is observed in response to females than males when bulk calcium signals are

measured by fiber photometry

. This ~2:1 female tuning bias is similar to that in MeApd

and MPOA

, but is inverted in VMHvl

. Thus, at each node in the processing pathway

from MeApd

→

BNSTpr

→

MPOA/VMHvl, intruder sex is represented as a cell identity code

(Extended Data Fig. 3i).

Yang et al.

Page 6

Nature

. Author manuscript; available in PMC 2023 February 03.

Author Manuscript

Second, while BNSTpr

Esr1

neuronal activity represents intruder sex identity, this activity

is not required to identify conspecific sex

. Although we confirmed that BNSTpr

Esr1

neuronal activity is required for males to show a preference for female cues

, we find

surprisingly that it is not required for them to identify and distinguish females vs males:

emission of USVs is only observed in response to female and not male cues, as in control

males

. Consistent with this, intruder sex can be decoded equally well from MPOA

Esr1

or VMHvl

Esr1

neural activity in the presence or absence of BNSTpr

Esr1

activity. While

the ability to identify and distinguish females vs. males is clearly a prerequisite for a

male to display a preference for females, it does not follow that all deficits in female

preference

necessarily reflect deficient sex identification. Our results call into question the

widespread use of male vs female preference assays as a surrogate measure of conspecific

sex identification.

Interestingly, female-biased conspecific sex-tuned neural representations have been observed

in the prefrontal cortex as well, where they play a role in sex preference behavior

. The

causal relationship between conspecific sex tuning in cortical vs. subcortical regions remains

to be determined. Our results also raise the question of whether the weak male-biased sex-

tuning observed in MeApd

in female mice extends to the structures we have investigated

here. Previous bulk calcium imaging studies suggest that BNSTpr AB neurons do not

encode intruder sex in females

, but single-cell analysis was not performed.

Third, while it may be intuitive to think that MeApd, BNSTpr and the hypothalamus each

play distinct functional roles in the transformation of sex-specific social cues into behavior,

such as social cue representation, sex identification and action selection, respectively, our

data on the transformations in neural representations that occur as information flows through

this circuit do not support such a strict hierarchical view. Instead, we find that the relative

variance explained by intruder sex identity vs. behavior is different in each of these regions.

In MPOA, the variance explained by behavior is higher than that in BNSTpr (and in MeApd)

(Fig. 5g, m and Extended Data Fig. 3i), consistent with a gradual shift in the relative

representations of sex identity vs. behavior from the amygdala to the hypothalamus. In

contrast, the variance explained by intruder sex in VMHvl is higher than that in BNSTpr,

largely because VMHvl has very few neurons tuned to specific phases of social behavior

Rather, the main transformation that occurs from MeApd and BNSTpr to VMHvl is

an inversion of the ratio of sex-tuned neurons, from female-dominant to male-dominant,

respectively (Fig. 5m).

Surprisingly, when BNSTpr

Esr1

neurons were silenced chemogenetically, the male sex-

tuning bias in VMHvl flipped to a female-tuning bias. Apparently, female tuning-dominant

Esr1

GABAergic neurons in BNSTpr are required to suppress female-specific tuning

among VMHvl

Esr1

glutamatergic neurons. Consistent with this, retrograde imaging indicated

that female-biased tuning is present among BNSTpr

Esr1

neurons that project to VMHvl

(as well as to MPOA; Extended Data Fig. 9). The mechanism whereby the female-biased

sex representation in BNSTpr is transformed into a male-biased representation in VMHvl

remains to be established, and may involve differential male- or female-biased inputs from

other regions, such as the posterior amygdala

–

. In contrast, silencing BNSTpr did

not significantly alter the female-dominant sex bias in MPOA (if anything it was slightly

Yang et al.

Page 7

Nature

. Author manuscript; available in PMC 2023 February 03.

Author Manuscript

increased); rather it reduced the variance explained by behavior (Fig. 5g). Together, these

data indicate a requirement for BNSTpr

Esr1

neuronal activity in shaping the representations

of social behavior and conspecific sex in MPOA and VMHvl, respectively, albeit to different

extents (Fig. 5m).

Finally, our results indicate that the inhibition of mounting and attack behavior that occurs

upon BNSTpr

Esr1

silencing is not secondary to a deficiency in sex identification

. Rather,

the failure to transition from appetitive to consummatory social behavior following silencing

reflects changes in the pattern of sex- and behavior-specific tuning among VMHvl

Esr1

and MPOA

Esr1

neurons, with no loss of sex identity coding. In what follows we suggest

plausible explanations for how these circuit-level “mutant phenotypes” may lead to the

observed behavioral phenotypes. However, this causality is not directly demonstrated, and

more complex explanations are certainly possible.

In MPOA, many Esr1

neurons are tuned for specific behaviors, including sniffing or

mounting (Fig. 5a and Extended Data Fig. 3i). Accordingly, as males transition from

sniffing to mounting, sniff-selective neurons become less active while mount-selective

neurons become more active. When BNSTpr

Esr1

neurons are silenced, this change in

MPOA neural representations of social behavior is muted: sniff-selective neurons continue

to be active (on the silenced side) as the animal transitions to mounting, while mount-

selective neurons exhibit reduced activity (Fig. 5a–e). Since ~95% of BNSTpr neurons are

GABAergic

, these observations suggest that in normal animals during sniffing,

sniff-selective neurons (or another MPOA subpopulation) may inhibit mount-selective

neurons. In order to transition to mounting, input from BNSTpr

Esr1

GABAergic neurons

may be required to inhibit the sniff-selective population, thereby dis-inhibiting the mount-

selective cells (Fig. 5n). In the absence of BNSTpr

Esr1

activity, mount-selective cells remain

inhibited and so this transition is blocked; instead, the male continues to sniff the female

(Extended Data Fig. 1b,

lower

In VMHvl, sniff- and attack-active cells are largely overlapping (Fig. 5i) but are sex-

specific

, with male-tuned neurons outnumbering female-tuned cells by ~2:1. During

attack, primarily male-tuned neurons are active

. Optogenetic experiments

and

computational modeling

suggest that the activity of VMHvl

Esr1

neurons must reach a

certain threshold in order for males to transition from sniff to attack. In the absence of

BNSTpr

Esr1

activity female-tuned neurons become dominant in VMHvl, so the number

and activity of male-selective neurons are reduced (Fig. 4c, Extended Data Fig. 8a).

This reduction may prevent activity in the male-tuned population from reaching the

threshold necessary to transition to attack (Fig. 5j–l, n), although increased inhibition from

neighboring GABAergic neurons may contribute as well. Together these data suggest that

activity in BNSTpr

Esr1

neurons controls sex- and behavior-tuning in MPOA and VMHvl in a

manner essential for the transition from sniffing to mounting or attack, respectively.

Functional perturbations and observational studies often yield different views of brain

function

. Our results demonstrate how calcium imaging can be combined with

manipulations of neural activity in freely moving animals to identify and link circuit-level to

behavioral phenotypes. This combined approach allows an integration of observational and

Yang et al.

Page 8

Nature

. Author manuscript; available in PMC 2023 February 03.

Author Manuscript

perturbational data to reveal both the transformations in neural representations that occur

as information flows through a circuit, and the potential function of such transformations

in the control of behavior. Future studies should reveal the detailed cellular and synaptic

mechanisms that underlie these important transformations.

METHODS

Mice

All experimental procedures involving the use of live mice or their tissues were carried

out in accordance with NIH guidelines and approved by the Institute Animal Care and

Use Committee (IACUC) and the Institute Biosafety Committee (IBC) at the California

Institute of Technology (Caltech). All C57BL/6N mice used in this study, including wild-

type and transgenic mice, were bred at Caltech. BALB/c male and female mice were used

as intruder mice and bred at Caltech or purchased from Charles River Laboratories (CRL).

Experimental mice were used at the age of 2–3 months. Intruder mice were used at the age

of 2–6 months and were maintained with three to five cage mates to reduce their aggression.

Esr1

Cre/+

knock-in mice (Jackson Laboratory stock no. 017911) were backcrossed into the

C57BL/6N background (>N10) and bred at Caltech. Heterozygous

Esr1

Cre/+

, or double

heterozygous

Aromatase

Cre/+

(ref.

);

Ai148/+

mice were used for cell-specific targeting

experiments and were genotyped by PCR analysis using genomic DNA from tail or ear

tissue. All mice were housed in ventilated micro-isolator cages in a temperature-controlled

environment (median temperature 23 °C, humidity 60%), under a reversed 11-h dark–13-h

light cycle, with ad libitum access to food and water. Mouse cages were changed weekly.

Virus

The following AAVs along with the supplier, injection titres in viral genome copies/ml

(vg/ml) and injection volumes in nanoliters (nl) were used in this study.

AAV1-hSyn1-SIO-stGtACR2-FusionRed (Addgene 105677, ~2 × 10

vg/ml, 200nl),

AAV5-hSyn-DIO-hM4D(Gi)-mCherry (Addgene 44362, ~2 × 10

vg/ml, 200nl per

injection), AAV5-Ef1a-DIO-eNpHR3.0-eYFP (Halo-eYFP, E.D. Fig. 2) (Addgene 26966,

~4 × 10

vg/ml, 200nl per injection), AAV1-syn-FLEX-jGCaMP7f-WPRE (Addgene

104492, ~2 × 10

vg/ml, 200nl per injection), AAVdj-syn-FLEX-FlpO (Janelia Vector

Core, ~1 × 10

vg/ml, 200nl per injection), AAVretro-syn-fDIO-jGCaMP7f (Janelia Vector

Core, ~2 × 10

vg/ml, 200nl per injection), AAVretro-CBH-DIO-nls-mScarlet (Janelia

Vector Core, ~1 × 10

vg/ml, 200nl per injection), AAVretro-CBH-DIO-nls-mNeonGreen

(Janelia Vector Core, ~1 × 10

vg/ml, 200nl per injection).

Stereotaxic Coordinates for virus injection and GRIN lens implantation

Stereotaxic injection coordinates were based on the Paxinos and Franklin atlas

Virus injection: BNSTpr, AP: −0.15, ML: ± 0.65, DV: −3.75, VMHvl, AP: −1.5, ML: ±

0.75, DV: −5.75, MPOA, AP: 1.0, ML: ± 0.3, DV: −4.75, MeApd, AP: −1.65, ML: ± 2.1,

DV: −4.9).

Gradient Refractive Index (GRIN) lens implantation:

Yang et al.

Page 9

Nature

. Author manuscript; available in PMC 2023 February 03.

Author Manuscript

BNSTpr: AP: −0.1, ML: −0.7, DV: −4.5 (

⌀

0.85 × 7mm GRIN lens with prism,

Extended Data Fig. 3).

BNSTpr: AP: −0.2, ML: −0.7, DV: −3.65 (

⌀

0.6 × 7.3mm GRIN lens).

MeApd: AP: −1.65, ML: −2.0, DV: −4.75(

⌀

0.6 × 7.3mm GRIN lens).

VMHvl: AP: −1.6, ML: −0.75, DV: −5.6 (

⌀

0.6 × 7.3mm GRIN lens).

MPOA: AP: +1.0, ML: −1.0, DV: −5.0 (

⌀

0.6 × 7.3mm GRIN lens). Implanted at

a 10° tilt in AP axis and a 5° tilt in ML axis to avoid damaging BNSTpr and its

projections to MPOA (Extended Data Fig. 5a–j).

Surgeries

Surgeries were performed on socially and sexually experienced adult male

Esr1

Cre/+

mice, aged 2–3 months. Virus injection and implantation were performed as described

previously

. Briefly, animals were anaesthetized with isoflurane (5% for induction and

1.5% for maintenance) and placed on a stereotaxic frame (David Kopf Instruments). Virus

was injected into the target area using a pulled glass capillary (World Precision Instruments)

and a pressure injector (Micro4 controller, World Precision Instruments), at a flow rate of

50 nl/minute. The glass capillary was left in place for 5 minutes following injections before

withdrawing. For microendoscope experiments, virus injection and lens implantation were

performed two weeks apart. Lenses were slowly lowered into the brain and fixed to the skull

with dental cement (Metabond, Parkell). 2 weeks after lens implantation, mice were head-

fixed on a running wheel and a miniaturized microendoscope (nVista, Inscopix) was lowered

over the implanted lens until GCaMP-expressing fluorescent neurons were in focus. Once

GCaMP-expressing neurons were detected, a permanent baseplate was attached to the skull

with dental cement. Mice were habituated with weight-matched dummy microendoscopes

(Inscopix) for at least 1 week before behavior testing.

Housing conditions for social and sexual experience

All male C57BL/6N mice used in this study were socially and sexually experienced. Mice

aged 8–12 weeks were initially co-housed with a female BALB/c for 1 week and were

screened for sex-appropriate social behaviors. Mice that show both mounting toward females

and attack toward males during a 30-minute resident intruder assay were selected for

surgery and subsequent behavior experiments. From this point forward, these male mice

were always cohoused with non-pregnant female BALB/c mice.

Chemogenetic inhibition

Behavioral tests were performed on two consecutive days. The number of mice receiving

saline or Clozapine-N-oxide (CNO) (Enzo Life Sciences) was counterbalanced across

the two days. CNO was dissolved in saline. CNO (7.5mg/kg) or saline (control) was

intraperitoneally injected 60 minutes prior to behavioral tests.

Optogenetic inhibition

Animals were connected to a 470nm or a 590nm laser (Changchun New Industries

Optoelectronics Tech Co., Ltd.) via optical patch cords (

⌀

200 μm, N.A., 0.22, Doric lenses

Yang et al.

Page 10

Nature

. Author manuscript; available in PMC 2023 February 03.

Author Manuscript

and Thorlabs) and a rotary joint (Doric lenses). The experimenter monitored mouse behavior

via a computer monitor in a room adjacent to the behavioral arena. Laser was triggered

manually when animals were engaged in the behaviors of interest. Sham stimulation (laser

OFF) was interleaved with the light stimulation (laser ON) as an internal control. For

optogenetic inhibition of BNSTpr using gtACR2, 10 seconds of 470nm continuous photo

stimulation at 1mW/mm

at the tissue level was used during social interactions. 10 minutes

of 470nm continuous photo stimulation at 1mW/mm

at the tissue level was used for urine

preference and “pencil-cup” (meshed barrier) tests. For optogenetic inhibition of BNSTpr

– MPOA and BNSTpr – VMHvl terminals using Halorhodopsin, 5 seconds of 590nm

continuous photo stimulation at 3mW/mm

at the tissue level was used.

Behavior annotations

Behavior videos were manually annotated using a custom MATLAB-based behavior

annotation interface

. A “baseline” period of 5 minutes where the animal was alone

in his home cage was recorded at the start of every recording session. A total of 7

behaviors during the resident intruder assays were annotated: sniff (face, body, genital-

directed sniffing) toward male or female intruders, attack, mount, intromission, “interact”

(periods where the animals were close to each other but sniff, attack, mount/intromission

were absent) with male or female intruders. For “dangled” presentations of male or female

intruders, the dangled bout starts at the onset of nostril/whiskers movement when the ano-

genital region of the dangled intruder is held next to the resident.

Microendoscopic imaging

On the day of imaging, mice were habituated for at least 10 minutes after installation of

the miniscope in their home cage prior to the start of the behavior tests. Imaging data was

acquired at 30 Hz with 2x spatial down-sampling, LED power (0.1–0.5) and gain (1x-7x)

were adjusted depending on the brightness of GCaMP expression determined by the image

histogram as per the user manual. A TTL pulse from the Sync port of the data acquisition

box (DAQ, Inscopix) was used to synchronously trigger StreamPix7 (Norpix) for video

recording. Imaging sessions typically lasted 1 hour (20–25 minutes of interactions per sex.)

ChemoScope imaging

To minimize population activity changes associated with stress from i.p. injections, social

experience or fatigue from resident intruder assays in saline or CNO trials, a series of

habituation imaging sessions were performed every 2–3 days (3–4 sessions total) prior to the

saline vs CNO trials used for analysis (Extended Data Fig. 5p–s). Mice were injected with

100μl of saline (or 100μl of 7.5mg/kg CNO for CNO trials) and returned to their home cage

for 60 minutes prior to imaging.

Microendoscopic data extraction

Preprocessing—

Miniscope data were acquired using the Inscopix Data Acquisition

Software as 2x down-sampled .isxd files. Preprocessing and motion correction was

performed using the Inscopix Data Processing Software. Briefly, raw imaging data was

cropped, 2x down-sampled, median filtered and motion corrected. A spatial band-pass filter

Yang et al.

Page 11

Nature

. Author manuscript; available in PMC 2023 February 03.

Author Manuscript

was then applied to remove out of focus background. The filtered imaging data was temporal

down-sampled to 10Hz and exported as a .tiff image stack.

Calcium data extraction—

After preprocessing, calcium traces were extracted and

deconvolved using the CNMF-E large data pipeline

with the following parameters:

patch_dims = [42, 42], gSig = 3, gSiz = 13, ring_radius = 19, min_corr = 0.8, min_pnr

= 8, deconvolution: foopsi with the ar1 model

. The spatial and temporal components of

every extracted unit were carefully inspected manually (SNR, PNR, size, motion artifacts,

decay kinetics, etc.) and outliers (obvious deviations from the normal distribution) were

discarded. The extracted traces were then Z-scored prior to analysis.

Choice probability

Choice probability (CP) is a metric that estimates how well either of two different

behaviors can be predicted/distinguished, based on the activity of any given neuron during

these two behaviors

. CP of single neurons were computed using previously described

methods

. To compute the CP of a single neuron for any behavior pair, 1-second binned

neuronal responses occurring during each of the two behaviors were used to generate a

“receiver operating characteristic” (ROC) curve. CP is defined as the area under the curve

(AUC) bounded between 0 to 1. A CP of 0.5 indicates the activity of the neuron cannot

distinguish between the two alternative behaviors. We defined a neuron as being capable

of distinguishing between two behaviors (e.g. interacting with male vs. female) if the

CP of that neuron was > 0.65 or < 0.35, and was > 2

or < −2

of the CP computed

using shuffled data (repeated 100 times). For instance, if the CP of a particular neuron for

females was 0.8, and the shuffled CP was 0.5±0.1 (SD), that neuron would be considered as

“female-preferring”. If another neuron had a CP of 0.1 and a shuffled CP of 0.5±0.15 (SD),

that neuron would be considered as a “male-preferring” neuron.

Decoding intruder sex

We constructed “frame-wise” (using 1-second binned behavior and calcium activity

“frames”, Fig. 3l) or “time-evolving” (as a function of time at 10Hz, Fig. 2i and Fig.

3m) linear SVM decoders (as described previously

) to distinguish intruder sex using 50

randomly selected neurons from each animal in BNST (n=15 animals), VMHvl (n=7) and

MPOA (n=7). Accuracy was evaluated using a stratified 5-fold cross-validator. Decoding

was repeated 100 times and the decoder performance was reported as the mean accuracy

per imaged animal. For significance testing, the mean accuracy of the decoder trained on

shuffled data (repeated 500 times per imaged animal) was computed to compare against the

decoder accuracy trained on actual data.

Determining explained variance in BNSTpr population activity

We calculated the variance of the population activity explained by intruder sex and behaviors

(Fig. 2o) using linear regression as described previously

. To evaluate the encoding of

intruder sex and behaviors by BNSTpr

Esr1

neurons, we regressed the activity of single

neurons against the sex of the intruder (a pair of binary vectors) and against 7 annotated

behaviors (7 binary vectors). The fraction of variance explained is calculated as the ratio of

the cross-validated R

(coefficient of determination) of the fit against the sex of the intruder

Yang et al.

Page 12

Nature

. Author manuscript; available in PMC 2023 February 03.

Author Manuscript

and the cross-validated R

of the fit against 7 different behaviors. To determine the fraction

of variance explained by behaviors, we regressed the residual activity (from the fit against

intruder sex) against male- or female- directed behaviors and reported the fraction of the

residual variance explained, as cross-validated R

of the fit.

Pearson’s correlation

Pearson’s correlation (Fig. 3j) was reported as the mean pair-wise Pearson’s correlation

coefficient per imaged animal computed by using the average population responses (all trials

during 1 recording session) to dangled male intruders and the 1-second binned population

responses to dangled male intruders (same-sex) or dangled female intruders (opposite-sex).

Mahalanobis distance ratio

Mahalanobis distance ratio (Fig. 3k) was expressed as the ratio between the Mahalanobis

distance (described previously

) of the population vector responding to dangled female

intruders (opposite sex) and the Mahalanobis distance of the population vector responding to

dangled male intruders (same sex).

Supervised UMAP embedding of single-unit responses to male and female stimuli

The first 50 PCs of 1-second-binned single-unit responses to male and female stimuli

from MPOA and VMHvl along with categorical labels indicating the origin of each unit

(MPOA pre-CNO, MPOA CNO, VMHvl pre-CNO, VMHvl CNO) were used to initialize

the embedding (Fig. 3q). The Python implementation of UMAP

(umap-learn) were used

along with the following parameters: n_neighbors = 15, min_dist = 0.4, metric = euclidean.

Statistical analysis

Data were processed and analyzed using Python, MATLAB, and GraphPad (GraphPad

PRISM 9). All data were analyzed using two-tailed non-parametric tests except urine

preference and pencil-cup (meshed barrier) tests (Extended Data Fig. 1r, w), and the

fraction of variance explained by intruder sex or female-directed behavior (Fig. 5g) where

two-way ANOVA with Bonferroni correction was used. Wilcoxon signed-rank test (paired,

non-parametric Mann-Whitney U test) was used for binary paired samples. Friedman test

with Dunn’s correction was used for non-binary paired samples (Fig. 5d, f). Kolmogorov-

Smirnov test was used for non-paired samples. Not significant (N.S.), P > 0.05; *P < 0.05;

**P < 0.01; ***P < 0.001; ****P < 0.0001.

Code availability

The custom MATLAB and Python codes used to analyze the data in this study are available

upon request.

Yang et al.

Page 13

Nature

. Author manuscript; available in PMC 2023 February 03.

Author Manuscript

Extended Data

Extended Data Figure 1: Chemogenetic and optogenetic silencing of BNSTpr

Esr1

neurons

disrupts aggression and mating.

, Illustration of hM4d (Gi)-mediated silencing in bilateral BNSTpr

Esr1

neurons.

, Example

behavior raster plot from 1 animal across 4 recording sessions.

, Percent of BNSTpr

Esr1

neurons that express mCherry. Values are plotted as mean ± SEM, n=6 mice.

d-m

Measurements performed before (pre-CNO: saline injection, gray points) vs. after (CNO

injection, maroon points) chemogenetic silencing of BNSTpr, of time spent sniffing male

intruders (

), number of attack bouts toward male intruders (

), time spent attacking male

intruders (

), time spent sniffing female intruders (

), number of mount bouts toward

female intruders (

), time spent mounting female intruders (

), time spent mounting male

intruders (

), time spent attacking female (

), and time spent attacking male (

), time spent

mounting female (

) in mCherry/CNO only controls (

l, m)

per 30-minute session. 100μl

Yang et al.

Page 14

Nature

. Author manuscript; available in PMC 2023 February 03.

Author Manuscript

containing either saline (pre-CNO) or 7.5mg/kg CNO was given i.p. 60 minutes prior to

start of behavior tests.

, Illustration of bilateral optogenetic silencing in BNSTpr.

o, t

Mean probability of sniff behavior occurring relative to onset of optogenetic silencing.

, Duration of sniffing toward male (

) or female (

) intruders.

q, v

, Duration of attack

(

) toward male intruders or mount (

) toward female intruders as a percentage of the

duration of optogenetic silencing. “Sham” controls were the same animals during a 10s

“sham stimulation,” i.e., without light.

r, w

, Percent of time spent interacting with male

or female urine (r), or male or female conspecifics separated by a meshed barrier (

)

before and during (

) or before, during and after (

) optogenetic silencing of BNSTpr.

Optogenetic silencing conditions: 470nm @ ~1 mW/mm

for 10 seconds (

o, p, q, t, u, v

470nm @ ~1 mW/mm

for 10 minutes (

r, s, w, x

) Statistics: For Chemogenetic inhibition,

two-sided Wilcoxon signed-rank test (

d-m

). Values are plotted as total time or total bouts

per 30-minute session per animal. n = 7 mice (

d-k

), n=4 mice (

l, m

). For Optogenetic

inhibition during interactions with male or female conspecifics (

o, p, q, t, u, v

), two-sided

Kolmogorov-Smirnov test (

o, t

), two-sided Wilcoxon signed-rank test (

p, q, u, v

). Male-

male trials (

o, p, q

), n=131 stim and 125 sham trials pooled from 10 mice. Male-female

trials mice (

t, u, v

), n=181 stim and 170 sham trials pooled from 10 mice. For Optogenetic

inhibition during urine preference (

) and “pencil cup” (mesh barrier) (

) tests, two-way

ANOVA with Bonferroni correction, n=6 mice. Values are plotted as mean ± SEM. ****p <

0.0001; ***p < 0.001; **p < 0.01; *p < 0.05. The mouse brain in this figure (

a, n

) has been

reproduced with permission from “The mouse brain in stereotaxic coordinate”, Paxinos G.

Yang et al.

Page 15

Nature

. Author manuscript; available in PMC 2023 February 03.

Author Manuscript

Extended Data Figure 2: Optogenetic silencing of BNSTpr

Esr1

terminals in MPOA or VMHvl

during male-male/ male-female interactions.

a, c

, Diagram illustrating BNST terminal silencing in MPOA (

) or VMHvl (

b, d

Representative images showing Halo-EYFP cell body expression in BNSTpr and axonal

expression in MPOA (

) and VMHvl (

). Similar expression patterns were observed in all

14 animals tested (

b, d

e, h, k, n

, Raster plots showing distribution of social behaviors

relative to optogenetic silencing of BNSTpr terminal in MPOA (

e, k

) or VMHvl (

h, n

f, i,

l, o

, Average probability of behavior occurring relative to onset of optogenetic silencing.

g, j,

m, p

, Duration of attack (

g, j

) or mount (

m, p

) as a percentage of the 5-second optogenetic

silencing. Statistics: two-sided Kolmogorov-Smirnov test (

f, i, l, o

), values are plotted as

mean ± SEM. Two-sided Wilcoxon signed-rank test (

g, j, m, p

). ****p < 0.0001; ***p <

0.001; **p < 0.01; *p < 0.05. n=7 mice for each silenced region. The mouse brain in this

figure (

a, c

) has been reproduced with permission from “The mouse brain in stereotaxic

Yang et al.

Page 16

Nature

. Author manuscript; available in PMC 2023 February 03.

Author Manuscript

Extended Data Figure 3: Performance of 0.85 mm Ø prism coupled GRIN lens compared to 0.6

mm Ø GRIN lens for imaging BNSTpr.

a, b, d, e

, Illustrations showing GRIN lens placement in BNSTpr from top

(a, d)

and side

(b,

views

. a-c

, Illustrations or data from animals implanted with prism lenses.

d-f

, Data from

animals implanted with conventional cylindrical GRIN lenses.

c, f

, Mean pixel correlation

during 1 example imaging session

. g

, Cumulative fraction of number of units captured per

imaging session normalized to the diameter of the GRIN lens.

, Cumulative fraction of

the peak to noise ratio (PNR) of all units imaged using either a 0.6mm grin lens or a

0.85mm prism-coupled grin lens.

, Raster plots of MeApd, BNSTpr, MPOA and VMHvl

Esr1

neuronal activity during male-male or male-female unrestrained social interactions.

For comparative purposes, all frames containing each behavior scored (indicated below

plot) during a 30 minute social interaction were concatenated and binned into 10s intervals;

Averaged z-scored responses of each unit across all bins are shown.

Yang et al.

Page 17

Nature

. Author manuscript; available in PMC 2023 February 03.

Author Manuscript