of 27
Nussenbaum, Martin
et al
. eLife 2023;12:e84260. DOI: https://doi.org/10.7554/eLife.84260
1 of 27
Novelty and uncertainty differentially
drive exploration across development
Kate Nussenbaum
1†
, Rebecca E Martin
1†
, Sean Maulhardt
1,2
, Yi (Jen) Yang
1,3
,
Greer Bizzell- Hatcher
1
, Naiti S Bhatt
1
, Maximilian Koenig
1,4
, Gail M Rosenbaum
1,5
,
John P O'Doherty
6
, Jeffrey Cockburn
6
, Catherine A Hartley
1
*
1
New York University, New York, United States;
2
University of Maryland, College Park,
United States;
3
Temple University, Philadelphia, United States;
4
Leiden University,
Leiden, Netherlands;
5
Geisinger Health System, Danville, United States;
6
Caltech,
Pasadena, United States
Abstract
Across the lifespan, individuals frequently choose between exploiting known rewarding
options or exploring unknown alternatives. A large body of work has suggested that children may
explore more than adults. However, because novelty and reward uncertainty are often correlated,
it is unclear how they differentially influence decision-
making across development. Here, children,
adolescents, and adults (ages 8–27 years,
N
= 122) completed an adapted version of a recently
developed value-
guided decision-
making task that decouples novelty and uncertainty. In line with
prior studies, we found that exploration decreased with increasing age. Critically, participants of all
ages demonstrated a similar bias to select choice options with greater novelty, whereas aversion
to reward uncertainty increased into adulthood. Computational modeling of participant choices
revealed that whereas adolescents and adults demonstrated attenuated uncertainty aversion for
more novel choice options, children’s choices were not influenced by reward uncertainty.
Editor's evaluation
This is an important study that investigates changes in novelty-
seeking and uncertainty-
directed
exploration from childhood to adulthood. A wide age range of participants was tested using a well-
suited task and performance was analyzed using a sophisticated model-
based approach. The results
provide compelling evidence that age-
related changes in decision-
making are driven by attenuation
in uncertainty aversion in the presence of stable novelty-
seeking from childhood to adulthood.
Introduction
Across the lifespan, exploration increases individuals’ knowledge of the world and promotes the
discovery of rewarding actions. In some circumstances, exploring new options may yield greater
benefits than sticking to known alternatives, whereas in others, ‘exploiting’ known options may bring
about greater rewards. This trade-
off is known as the ‘explore–exploit’ dilemma (
Cohen et al., 2007
;
Sutton et al., 1998
), reflecting the challenge inherent to resolving this tension. In general, the optimal
balance between exploration and exploitation may shift across the lifespan. Relative to adults, chil-
dren tend to know less about the world and have longer temporal horizons over which to exploit
newly discovered information (
Gopnik, 2020
;
Gopnik et al., 2017
). Thus, it may be advantageous to
explore to a greater extent earlier in life, and gradually shift to a more exploitative decision strategy as
experience yields knowledge. Empirical data suggest that individuals at varied developmental stages
do indeed tackle explore–exploit problems differently. Children and adolescents tend to explore
more than adults (
Christakou et al., 2013
;
Giron et al., 2022
;
Jepma et al., 2020
;
Lloyd et al.,
RESEARCH ARTICLE
*For correspondence:
cate@nyu.edu
These authors contributed
equally to this work
Competing interest:
See page
15
Funding:
See page 15
Preprinted:
28 April 2022
Received:
17 October 2022
Accepted:
07 August 2023
Published:
16 August 2023
Reviewing Editor:
David Badre,
Brown University, United States
Copyright Nussenbaum,
Martin
et al
. This article is
distributed under the terms
of the Creative Commons
Attribution License, which
permits unrestricted use and
redistribution provided that the
original author and source are
credited.
Research article
Neuroscience
Nussenbaum, Martin
et al
. eLife 2023;12:e84260. DOI: https://doi.org/10.7554/eLife.84260
2 of 27
2021
;
Nussenbaum and Hartley, 2019
;
Schulz et al., 2019
), and this increased exploration promotes
enhanced learning about the structure of the environment (
Blanco and Sloutsky, 2021
;
Liquin and
Gopnik, 2022
;
Sumner et al., 2019
). Despite compelling arguments for why an early bias toward
exploration may be advantageous and growing evidence that children
are
in fact more exploratory
than adults, the cause of the developmental shift toward exploitation remains unclear.
Prior work has revealed that across development, two features of choice options influence explo-
ration: stimulus
novelty
(
Daffner et al., 1998
;
Gottlieb et al., 2013
;
Henderson and Moore, 1980
;
Jaegle et al., 2019
;
Kakade and Dayan, 2002
;
Wittmann et al., 2008
) and reward
uncertainty
(
Badre et al., 2012
;
Blanco and Sloutsky, 2021
;
Gershman, 2018
;
Somerville et al., 2017
;
Trudel
et al., 2021
;
Wang et al., 2021
;
Wilson et al., 2014
). Here, we use
novelty
to refer to the extent to
which choice options have been previously encountered and
uncertainty
to refer to the variance in the
distributions of rewards that they yield. Disentangling the role of novelty and uncertainty in driving
exploratory decision making is challenging because they are often correlated. For example, a new
toy has high novelty because it has never been encountered and high reward uncertainty because its
entertainment value is unknown. Still, while novel stimuli almost always have high reward uncertainty,
in many cases,
familiar
options do as well — when buying a familiar toy as a gift for
someone else
, one
may have little knowledge of how much they will like it.
A recent study in adults took advantage of these types of choices, and, by harnessing familiar
options with unknown reward probabilities, decoupled the influence of novelty and uncertainty on
exploratory decision making in adults (
Cockburn et al., 2022
). Adults were novelty-
seeking, pref-
erentially selecting choice options that they had encountered infrequently in the past versus those
that were more familiar. However, adults were also uncertainty averse, such that they tended to avoid
options with high reward uncertainty. This tension between avoiding uncertain options while pursuing
novel alternatives, which are themselves inherently uncertain, suggested
interactive
effects of choice
features. Computational modeling further revealed that stimulus novelty diminished the influence of
uncertainty on exploratory choice. Thus, these findings suggest that value-
guided decision making in
adults — and specifically, the balance between exploration and exploitation — may be governed by
complex interactions among different features of choice options. To date, however, the influences of
novelty and uncertainty have not been disentangled in children and adolescents.
Changes in the influence of these choice features may shift the explore–exploit balance across
development. A stronger appetitive influence of stimulus novelty may drive greater exploration earlier
in life. Reduced uncertainty aversion, or perhaps even an early preference to explore more uncertain
options, may similarly promote heightened exploratory behavior. Novelty and uncertainty may also
exert unique, interactive effects for younger individuals. Though prior studies have found effects of
novelty (
Henderson and Moore, 1980
;
Mendel, 1965
) and reward uncertainty (
Blanco and Sloutsky,
2021
;
Meder et al., 2021
;
Schulz et al., 2019
) on exploration and choice in early childhood, it is
unclear how their relative influence changes from childhood to early adulthood, leaving open the
question of
why
children tend to explore more than adults. Further, in most developmental studies,
novelty and uncertainty are confounded, making it difficult to tease apart their separate, motivational
effects.
Here, using an adapted version of the task introduced in
Cockburn et al., 2022
with a large
age-
continuous developmental sample, we asked how the influence of novelty and uncertainty on
exploratory choice changes from middle childhood to early adulthood. We hypothesized that the
developmental shift from more exploratory to more exploitative behavior would be driven by changes
in how both novelty and reward uncertainty affect the evaluation of choice options from childhood to
adulthood.
Results
Approach
Participants ages 8–27 years (
N
= 122; mean age = 17.9 years, standard deviation [SD] age = 5.6
years, 62 females, 59 males, 1 non-
binary) completed a child-
friendly decision-
making task adapted
from one used in a prior adult study (
Cockburn et al., 2022
). In the task, participants tried to find gold
coins that various creatures had hidden in different territories. The task was divided into 10 blocks
of 15 trials. Each block took place within a different territory in which coins were hidden by a distinct
Research article
Neuroscience
Nussenbaum, Martin
et al
. eLife 2023;12:e84260. DOI: https://doi.org/10.7554/eLife.84260
3 of 27
creature. Each creature hid their coins among three possible locations, one of which held a coin on
either 20 or 30% of trials (on easy and hard blocks, respectively), one of which held a coin on 50%
of trials, and one of which held a coin on either 70 or 80% of trials. Participants were not explicitly
informed of these probabilities, and had to learn, through trial and error, where each creature was
most likely to hide a coin. On every trial, participants viewed two hiding spots and had to select one
in which to search for a coin (
Figure 1
). After a brief delay, participants saw the outcome of their
choice — either a coin or an X indicating that they had not found a coin. Throughout each block, the
background of the screen indicated the territory and a picture in the lower left corner indicated the
creature that had hidden coins there.
Importantly, after the first block, each subsequent block contained two hiding spots that partic-
ipants had encountered in previous blocks and one novel hiding spot they had not seen before.
Though participants had already encountered two of the hiding spots within each block, the reward
probabilities were re-
randomized for every creature. In this way, the task dissociated sensory
novelty
and reward
uncertainty
. At the beginning of every block, the novelty of each hiding spot varied but
all
hiding spots had high reward uncertainty. Participants were explicitly told that the reward probabilities
were reset in every block; within the task narrative, this was framed as each creature having different
favorite hiding spots in their respective territory (see Appendix 1 for analyses demonstrating that
participants of all ages indeed comprehended these instructions and ‘reset’ the reward probabilities
at the beginning of each block). After the exploration task, participants completed a surprise memory
test in which they were shown each of the ten creatures, one at a time, and asked to select its favorite
hiding spot from an array of five options.
Exploration task performance
First, we examined whether participants learned to select the better options within each block of the
task. On each trial, the optimal choice was defined as the option with the higher reward probability.
A mixed-
effects logistic regression examining the effects of within-
block trial number, age, block diffi-
culty, block number, and their interactions, with random participant intercepts and slopes across trial
number, block difficulty, and block number revealed that participants learned to make more optimal
choices over the course of each block, odds ratio (OR) = 1.11, 95% confidence interval = [1.07, 1.16],
χ
2
(1) = 24.5,
p
< 0.001. In addition, participants learned faster, OR = 1.05 [1.01, 1.08],
χ
2
(1) = 8.2,
p
Figure 1.
Exploration task. Participants completed 10 blocks of 15 choice trials in which they selected between two of three ‘hiding spots’ to find gold
coins. Within each block, two hiding spots had been previously encountered and one was completely novel. Each block took place within a different
‘territory’ in which a new creature hid coins. Each creature had different preferred hiding spots, such that the reward probabilities associated with each
option were reset at the beginning of each block.
Research article
Neuroscience
Nussenbaum, Martin
et al
. eLife 2023;12:e84260. DOI: https://doi.org/10.7554/eLife.84260
4 of 27
= 0.004, and made more optimal choices in easy relative to hard blocks, OR = 1.11 [1.07, 1.15],
χ
2
(1)
= 25.5,
p
< 0.001 (
Figure 2A
). Performance also improved with increasing age, OR = 1.10 [1.03, 1.18],
χ
2
(1) = 7.1,
p
= 0.008 (
Figure 2A
). While we did not observe a main effect of block number, we did
observe a block number × block difficulty interaction, OR = 0.96 [0.93, 0.99],
χ
2
(1) = 7.3,
p
= 0.007,
as well as a block difficulty × trial × block number interaction, OR = 0.94 [0.91, 0.97],
χ
2
(1) = 15.1,
p
< 0.001, such that performance differences between easy and hard blocks were greater earlier in the
experiment. No other interactions reached significance (
p
s > 0.07).
We followed the same analytical approach to examine whether participants earned reward on each
trial, though we removed the block difficulty and block number random slopes to allow the model
to converge. Here, we similarly observed that performance improved across trials within each block,
OR = 1.03 [1.00, 1.07],
χ
2
(1) = 3.9,
p
= 0.049. Further, the rate at which participants learned to make
rewarding choices was faster in easier versus harder blocks, OR = 1.03 [1.00, 1.06],
χ
2
(1) = 3.9,
p
=
0.048, though the effect of block difficulty was greater in earlier blocks, OR = 0.97 [0.94, 1.00],
χ
2
(1)
= 4.0,
p
= 0.045. No other main effects or interactions reached significance.
Taken together, these findings indicate that participants across our age range learned to select
rewarding choice options throughout each block, though the extent to which participants learned
to ‘exploit’ the most rewarding choice options increased across age. Participants also demonstrated
above-
chance (defined as 0.2) memory for the most rewarding choice within each block (mean = 0.25;
standard error [SE] = 0.001;
t
(119) = 4.02,
p
< 0.001). Memory accuracy did not vary across age,
β
= −0.046, SE = 0.07,
z
= −0.69,
p
= 0.49, and did not significantly relate to individual differences in
learning (see Appendix 1).
Age-related change in exploration
Next, we turned to our main questions of interest: whether and how novelty and uncertainty influ-
enced exploration across age. To examine the influence of expected value, uncertainty, and stimulus
novelty on choice behavior, we defined and computed these three feature values for each choice
option on every trial (
Cockburn et al., 2022
). Expected value was defined as the mean of the beta
distribution specified according to the win and loss history of each choice option (hiding spot) within
the block:
α
α
+
β
where
α
= number of wins +1 and
β
= number of losses +1. Uncertainty was defined
as the variance of this beta distribution:
(
α
β
)
2
α
+
β
+1
. Stimulus novelty was determined by taking the vari-
ance of a different beta distribution, where
α
= the number of times participants had seen the choice
option before throughout the entire task +1 and
β
= 1.
To address how these choice features differentially influenced exploration across development,
we computed the differences in expected value, uncertainty, and novelty between the left and right
Figure 2.
Exploration task performance. (
A
) Participants’ (n = 122) proportion of optimal choices as a function of age and block difficulty. (
B
) Reward
participants earned in easy and hard blocks of the task across age. In both plots, points represent participant averages in each block condition, lines
show the best-
fitting linear regression modeling the effect of age, and the shaded regions around them represent 95% confidence intervals. The dotted
lines indicate chance-
level performance.
Research article
Neuroscience
Nussenbaum, Martin
et al
. eLife 2023;12:e84260. DOI: https://doi.org/10.7554/eLife.84260
5 of 27
choice options on every trial. We then ran a mixed-
effects logistic regression examining how these
differences — as well as their interactions with continuous age — related to the probability that the
participant chose the left option on every trial. Participants were more likely to select the options that
they had learned were more valuable, OR = 3.22 [2.82, 3.67],
χ
2
(1) = 155.8,
p
< 0.001. However,
expected value was not the only driver of choice behavior. Participants also demonstrated a bias
toward selecting more novel stimuli, OR = 1.34 [1.28, 1.41],
χ
2
(1) = 104.2,
p
< 0.001, and a bias
away
from choosing options with greater uncertainty, OR = 0.89 [0.83, 0.95],
χ
2
(
1
)
= 11.9,
p
< 0.001. On
trials in which the two choice options had similar-
expected-
values (<0.05 difference), participants
selected the more novel option on 58.7% (SE = 0.8%) of trials and the more uncertain option on only
46.2% (SE = 0.9%) of trials.
The influence of expected value, novelty, and uncertainty on choice behavior each followed distinct
developmental trajectories. Results from our regression model indicated that younger participants’
choices were less value-
driven relative to those of older participants, as reflected in a significant age
× expected value interaction, OR = 1.22 [1.07, 1.39],
χ
2
(
1
)
= 8.97,
p
= 0.003 (
Figure 3A
). These
findings are consistent with a broader literature that has observed age-
related improvements in the
computation of expected value (
Rosenbaum and Hartley, 2019
). Importantly, however, age-
related
increases in these ‘exploitative’ choices were
not
driven by age-
related differences in novelty-
seeking;
there was not a significant interaction between age and novelty, OR = 1.02 [0.98, 1.07],
χ
2
(1) = 0.96,
p
= 0.327. In contrast to the relative stability of this novelty preference across age, we observed a signif-
icant age × uncertainty interaction effect, OR = 0.89 [0.84, 0.95],
χ
2
(1) = 11.3,
p
< 0.001, indicating
greater uncertainty aversion in older participants (
Figure 3A
). All findings held when we included
block number and within-
block trial number as interacting fixed effects in the model (see Appendix 1).
We further examined whether age-
related increases in uncertainty aversion were due to an early
preference
to engage with uncertain options or early indifference to uncertainty. To test these possi-
bilities, we ran an additional mixed-
effects logistic regression including data only from child partici-
pants, examining how expected value, uncertainty, and novelty influenced choice. Results indicated
that children’s choices were significantly influenced by both expected value (OR = 2.27 [1.78, 2.91],
χ
2
(1) = 26.8,
p
< 0.001) and novelty (OR = 1.35 [1.24, 1.48],
χ
2
(1) = 28.4, p < 0.001). However, there
was not an effect of uncertainty on choice, OR = 1.07 [0.945, 1.21],
χ
2
(1) = 1.17,
p
= 0.280, indicating
no significant evidence for uncertainty-
seeking.
Corroborating these findings, on trials in which the two choice options had nearly identical
expected values (<0.05 difference), children, adolescents, and adults, on average, selected the more
novel option on 59.1% (SE = 1.7%), 59.1% (SE = 1.7%), and 58.3% (SE = 1.2%) of trials, respectively
(
Figure 3B
). However, whereas adults tended to avoid the more uncertain option, selecting it on only
Figure 3.
Influence of expected value, uncertainty, and novelty on choice behavior across age. (
A
) The proportion of all trials in which participants (n
= 122) chose the left, more novel, and more uncertain choice option as a function of the expected value difference between the options. Participants
were more likely to choose options with greater expected value, higher novelty, and lower uncertainty (
p
s < 0.001). The influence of novelty did not
vary across age, whereas uncertainty was more aversive in older participants (
p
< 0.001). Points indicate age group means and error bars show standard
errors. (
B
) The proportion of similar-
expected-
value trials (difference between the two options <0.05) in which participants chose the more novel and
more uncertain option, plotted as a function of continuous age. The lines show the best-
fitting linear regression lines and the shaded regions around
them represent 95% confidence intervals.
Research article
Neuroscience
Nussenbaum, Martin
et al
. eLife 2023;12:e84260. DOI: https://doi.org/10.7554/eLife.84260
6 of 27
41.9% (SE = 1.2%) of equal-
expected-
value trials, adolescents and children selected the more uncer
-
tain option on 48.9% (SE = 1.7%) and 52.7% (SE = 1.7%) of these trials, respectively (
Figure 3B
). Thus,
taken together, these results suggest that age-
related decreases in exploratory choices were driven
by an increase in aversion to reward uncertainty with increasing age.
Age-related change in sensitivity versus aversion to uncertainty
There are two potential accounts of the observed increase in uncertainty aversion with increasing age:
Children may have been sensitive to reward uncertainty but not averse to it, or, they may have failed
to track uncertainty at all due to the computational demands of estimating the variance of outcome
distributions across the task. To disentangle these two possibilities, we examined how the uncertainty
of the selected choice option influenced participant response times. If younger participants were
sensitive to reward uncertainty, their response times should relate to it.
A linear mixed-
effects model examining how the expected value, novelty, and uncertainty of the
selected choice option — as well as their interactions with age — related to log-
transformed response
times revealed sensitivity to all three choice features across age (
Figure 4
). Participants responded
more quickly when selecting options with higher expected values (
b
= −0.03, SE = 0.006,
F
(1, 115.4)
= 29.7,
p
< 0.001), and more slowly when selecting options with higher novelty (
b
= 0.06, SE = 0.004,
F
(1, 111.3) = 197,
p
< 0.001) and higher uncertainty (
b
= 0.07, SE = 0.005,
F
(1, 117.1) = 171.7,
p
<
0.001). We also observed significant novelty by age (
b
= −0.009, SE = 0.004,
F
(1, 108.2) = 4.2,
p
= 0.043) and uncertainty by age interactions (b = 0.015, SE = 0.005,
F
(1, 117.8) = 8.2,
p
= 0.005).
Novelty had a stronger slowing influence on the response times of younger versus older participants,
whereas uncertainty had a stronger slowing influence on the response times of older versus younger
participants (
Figure 4
). Critically, however, we continued to observe a significant effect of uncertainty
on response times when we only included children’s data in the model, b = 0.04, SE = 0.01,
F
(1, 38.2)
= 16.5,
p
< 0.001, indicating that children’s response times were sensitive to the uncertainty of the
selected option (see
Appendix 1—table 5
for regression coefficients for all choice features cross age
Figure 4.
Influence of expected value, uncertainty, and novelty on choice response times across age. (
A
) Participants (n = 122) were faster to select
options with higher expected values, (
B
) slower to select options with greater uncertainty, and (
C
) slower to select options with higher novelty. Individual
points in panels A–C show individual log-
transformed response times on each trial. The lines in panels A–C show the best-
fitting linear regression lines
and the shaded regions around them represent 95% confidence intervals. (
D
) The influence of expected value on response times did not vary across
age, whereas younger participants demonstrated (
E
) a weaker influence of uncertainty on response times (
p
= 0.005) and (
F
) a stronger influence of
novelty on response times (
p
= 0.043). The lines in panels D–E show predictions from a linear mixed-
effects model and the shaded regions around them
represent 95% confidence intervals.
Research article
Neuroscience
Nussenbaum, Martin
et al
. eLife 2023;12:e84260. DOI: https://doi.org/10.7554/eLife.84260
7 of 27
groups). These findings held when we controlled for within-
block trial number in the models (
p
s <
0.01).
We also examined whether slower response times reflected uncertainty aversion by examining the
relation between individual differences in the influence of uncertainty on choice and in the influence
of uncertainty on response times. To do so, we extracted individual participants’ random uncertainty
slopes from a model examining how expected value, novelty, and uncertainty differences between
choice options related to decisions, and individual participants’ random uncertainty slopes from our
model examining how the expected value, novelty, and uncertainty of the selected choice option
related to response times. We then ran a linear regression examining how the effect of uncertainty
on response times, age, and their interaction related to uncertainty aversion. We observed a signif-
icant negative relation between the effect of uncertainty on choice and the effect of uncertainty on
response times,
β
= −0.22, SE = 0.09,
t
= −2.5,
p
= 0.014, indicating that participants who were
more uncertainty averse were also slower to select more uncertain options. We did not observe a
significant age x uncertainty interaction effect (
p
= 0.81). Thus, taken together, our findings suggest
that despite not demonstrating uncertainty aversion in their decisions, children
were
sensitive to the
relative reward uncertainty of different choice options throughout the task.
Computational characterization of choice
As in
Cockburn et al., 2022
, we observed opposing effects of novelty and uncertainty on choice —
though participants sought out novel options, they shied away from those with greater uncertainty.
At first glance, these results are somewhat puzzling because novel options are
inherently
uncertain.
Reinforcement learning models that formalize different algorithms for how the expected utilities of
the choice options are computed across trials can provide greater insight into how novelty and uncer
-
tainty may
interact
to influence exploratory choice behavior.
We fit participant choice data with six different reinforcement learning models (see methods).
Across models, we conceptualized the learning process as that of a ‘forgetful Bayesian learner,’ such
that the expected value of each choice option is computed as the mean of a beta distribution with
hyperparameters that reflect recency-
weighted win and loss outcomes (
Cockburn et al., 2022
). We
then modified this baseline model by adding either fully separable or interacting uncertainty and
novelty biases. Specifically, beyond the baseline model, we fit three additional models in which uncer
-
tainty and novelty exerted separable influences on choice behavior: a model augmented with a
novelty
bias
that adjusted the initial hyperparameters of each option’s beta distribution, a model augmented
with an
uncertainty bias
that added or subtracted each option’s scaled uncertainty to its expected
utility, and a model augmented with both biases. Corroborating our behavioral results, parameter
estimates from the model with both a novelty and uncertainty bias revealed an age-
consistent novelty
preference but age-
varying uncertainty aversion (see Appendix 1).
We additionally fit two models that account for interactions between novelty and uncertainty.
Given the findings of
Cockburn et al., 2022
, we hypothesized that novelty may buffer the aversive
influence of reward uncertainty. In other words, we expected that the extent to which the uncertainty
of a given choice option would influence its utility would increase in relation to its familiarity. Thus, we
fit two additional models (with and without a separate novelty bias) in which the uncertainty bias was
‘gated’ by stimulus familiarity (though the model with both a novelty bias and familiarity gate was not
recoverable; see ‘methods’).
To test for age-
related change in the way that novelty and uncertainty influenced exploratory
choice, we compared model fits for these six models within each age group using a random-
effects
Bayesian model selection procedure with simultaneous hierarchical parameter estimation (
Piray et al.,
2019
) and examined protected exceedance probabilities (PXPs), which reflect the probability that a
given model in a comparison set is the most frequent, best-
fitting model across participants, while
controlling for differences in model frequencies that may arise due to chance.
In line with findings from
Cockburn et al., 2022
, we found that adult choices were best character
-
ized by a model in which choice utilities took into account
interactions
between novelty and uncer
-
tainty. Specifically, adult choices were best captured by the familiarity-
gated uncertainty model (PXP
Familiarity Gate = 1), in which uncertainty aversion was greater for more familiar options. Despite
showing weaker aversion to uncertainty relative to adults, adolescents were also best fit by this model
Research article
Neuroscience
Nussenbaum, Martin
et al
. eLife 2023;12:e84260. DOI: https://doi.org/10.7554/eLife.84260
8 of 27
(PXP Familiarity Gate = 1). Children’s choices, however, were best captured by a model with a simple
novelty bias (PXP Novelty Bias = 0.62; PXP Familiarity Gate = 0.38).
Parameter estimates from the winning models reflected participants’ bias toward novel stimuli
and away from those with high reward uncertainty. Children’s average ‘novelty bias’ (from the group-
level novelty bias model fits) was 1.49, indicating that they optimistically initiated the value of novel
options. A one-
sample hierarchical Bayesian inference (HBI)
t
-
test examining the group-
level posterior
distribution of the novelty bias parameter (implemented via the cbm model-
fitting package
Piray
et al., 2019
), revealed that children’s novelty bias was significantly different from 0,
t
(16.3) = 5.96,
p
< 0.001. The average value of the ‘uncertainty bias’ (from the group-
level familiarity-
gated uncer
-
tainty model fits) was −0.15 for both adolescents and adults. HBI
t
-
tests revealed that uncertainty bias
parameter estimates were significantly different from 0 in both age groups (Adolescents:
t
(26.5) =
−7.8,
p
< 0.001; Adults:
t
(11.3) = −4.16,
p
= 0.001).
Model simulations revealed that the winning models well-
captured qualitative features of behav-
ioral choice data for each age group. For each model, we generated 50 simulated datasets using each
of the 122 participants’ trial sequence and parameter estimates (for a total of 6100 simulated agents
per model). For each model, we then computed the performance of 122 simulated participants by
averaging the performance of the 50 agents who shared the same trial sequence. Data from these
simulations demonstrated that the familiarity-
gated uncertainty model generated the most strongly
diverging effects of novelty and uncertainty on choice, in line with the adult and adolescent data
(
Figure 5
; also see
Appendix 1—figure 2
). The simpler novelty bias model instantiated a bias toward
both
novel and uncertain choices. Thus, these modeling results suggest that whereas adults and
adolescents were more strongly deterred by the uncertainty of familiar options versus novel ones,
children employed a simpler learning algorithm in which they optimistically initialized the value of
novel choice options.
Discussion
In this study, we investigated how novelty and uncertainty influence exploration across development.
Though new choice options tend to have
both
high novelty and high reward uncertainty, we found
that the influence of these features on decision making follow distinct developmental trajectories.
While participants across age demonstrated a similar bias toward selecting more novel choice options,
only older participants showed aversion to selecting those with greater uncertainty. These findings
suggest that children’s bias toward exploration over exploitation may arise from attenuated aversion
to selecting more uncertain options rather than heightened sensitivity to novelty.
Prior studies have found that novelty may be intrinsically rewarding (
Wittmann et al., 2008
), moti-
vating individuals to approach, learn about, and remember the new stimuli they encounter (
Houillon
et al., 2013
;
Krebs et al., 2009
). Children (
Henderson and Moore, 1980
;
Mendel, 1965
;
Valenti,
1985
), adolescents (
Spear, 2000
), and adults (
Cockburn et al., 2022
;
Daffner et al., 1998
) all demon-
strate novelty-
seeking behavior. However, though many studies have shown novelty preferences at
Figure 5.
Model simulations. The average proportion of similar-
expected-
value trials (with expected value magnitude differences <0.05) in which
both real (n = 122) and simulated (n = 122) participants chose the more novel and more uncertain option. The shaded regions show the empirical data
and best-
fitting model for each age group. Error bars represented the standard error across participant means. The novelty bias and familiarity-
gated
uncertainty model each had three free parameters, while the baseline model had two, and the novelty + uncertainty bias model had four.
Research article
Neuroscience
Nussenbaum, Martin
et al
. eLife 2023;12:e84260. DOI: https://doi.org/10.7554/eLife.84260
9 of 27
different developmental stages, little work has compared novelty preferences across age. Research in
rodents has suggested that adolescents may demonstrate heightened sensitivity to novelty (
Philpot
and Wecker, 2008
;
Spear, 2000
;
Stansfield and Kirstein, 2006
), but to the best of our knowledge,
there is no human evidence for an adolescent peak in novelty preferences. Indeed, our findings suggest
that the drive to engage with novel stimuli promotes exploratory choice in a consistent manner across
our age range. Moreover, the consistency of novelty-
seeking across development indicates that differ
-
ences in the reward value assigned to novel options cannot fully account for developmental differ
-
ences in exploration.
Whereas novelty-
seeking did not exhibit age-
related change, uncertainty aversion increased from
childhood to early adulthood, potentially reflecting developmental improvement in the strategic
modulation of information-
seeking. Prior studies of decision making have found that individuals across
age demonstrate uncertainty aversion in some environments (
Camerer and Weber, 1992
;
Payzan-
LeNestour et al., 2013
;
Rosenbaum and Hartley, 2019
) and uncertainty-
seeking in others (
Blanchard
and Gershman, 2018
;
Giron et al., 2022
;
Schulz et al., 2019
). These seemingly discrepant patterns
of behavior may be explained by differences in the utility of resolving uncertainty across contexts. In
environments where learned information can be exploited to improve subsequent choices, resolving
uncertainty has high utility (
Rich and Gureckis, 2018
;
Wilson et al., 2014
), whereas in choice contexts
with short temporal horizons, there is little opportunity to use learned reward information to improve
future decisions (
Camerer and Weber, 1992
;
Levy et al., 2010
). In our task, individuals had a rela-
tively short horizon over which to exploit reward probabilities that themselves required multiple trials
to learn — children’s reduced uncertainty aversion may have emerged from insensitivity to the limited
utility of gaining additional information about the most uncertain choice options (
Somerville et al.,
2017
).
Importantly, even though children’s choices were not uncertainty averse, children’s slower response
times when engaging with more uncertain options suggests that they were able to track uncertainty.
Indeed, the developmental decrease in uncertainty-
seeking behavior that we observed here is in line
with what has been observed in prior studies in which uncertainty was easier to discern. For example,
in one study (
Blanco and Sloutsky, 2021
), children tended to select choice options with hidden
reward amounts over those with visible reward amounts. In other studies with spatially correlated
rewards, children could use the layout of revealed outcomes to direct their sampling toward unex-
plored regions (
Giron et al., 2022
;
Meder et al., 2021
;
Schulz et al., 2019
). In studies of causal
learning and exploratory play, young children often use experiences of surprise or the coherence of
their own beliefs about the world to direct their exploration toward uncertain parts of the environment
(
Bonawitz et al., 2012
;
Schulz and Bonawitz, 2007
;
Wang et al., 2021
). Here, we extend these past
findings to demonstrate that children are sensitive to uncertainty even when it depends on distribu-
tions of binary outcomes.
Our observation of an influence of uncertainty on children’s reaction times suggests that uncer
-
tainty did affect how children made value-
based decisions. Future work could fit cognitive models to
both participants’ choices and response times to investigate how, across age, uncertainty influences
component cognitive processes involved in value-
based decision making. For example, researchers
could use sequential sampling models to test different hypotheses about how value uncertainty —
and its interactions with both expected value and novelty — influences both the rate at which partici-
pants accumulate evidence for a given option as well as the evidence threshold that must be reached
for a response to be made (
Lee and Usher, 2023
;
Wu et al., 2022
). In addition, these approaches
could be integrated with reinforcement learning models (
Fontanesi et al., 2019
) to gain further reso-
lution into how the learned features of different options influence the choices participants make and
the speed with which they make them.
Our computational modeling findings largely replicated previous work suggesting that by adult-
hood, novelty and uncertainty interact competitively (
Cockburn et al., 2022
), exerting opposing
motivational influences on decision-
making. Using functional magnetic resonance imaging (fMRI), this
prior adult study (
Cockburn et al., 2022
) revealed that activation in the ventral striatum reflects a
biased reward prediction error consistent with optimistic value initialization for novel stimuli, whereas
activation in the medial prefrontal cortex (mPFC) reflects the subjective utility of uncertainty reduction.
In line with choices being best characterized by a model in which the aversive influence of uncer
-
tainty was dampened by novelty, the integration of uncertainty into these mPFC value representations