CaltechAUTHORS
  A Caltech Library Service

Novelty and uncertainty interact to regulate the balance between exploration and exploitation in the human brain

Cockburn, Jeffrey and Man, Vincent and Cunningham, William and O'Doherty, John P. (2021) Novelty and uncertainty interact to regulate the balance between exploration and exploitation in the human brain. . (Unpublished) https://resolver.caltech.edu/CaltechAUTHORS:20211015-222205042

[img] PDF - Submitted Version
Creative Commons Attribution Non-commercial No Derivatives.

2MB

Use this Persistent URL to link to this item: https://resolver.caltech.edu/CaltechAUTHORS:20211015-222205042

Abstract

Recent evidence suggests that both novelty and uncertainty act as potent features guiding exploration. However, these variables are often conflated with each other experimentally, and an understanding of how these attributes interact to regulate the balance between exploration and exploitation has proved elusive. Using a novel task designed to decouple stimulus novelty and estimation uncertainty, we identify separable behavioral and neural mechanisms by which exploration is colored. We show that uncertainty was avoided except when the information gained through exploration could be reliably exploited in the future. In contrast, and contrary to existing theory, novel options grew increasingly attractive relative to familiar counterparts irrespective of the opportunity to leverage their consequences and despite the uncertainty inherent to novel options. These findings led us to develop a formal computational framework in which uncertainty directed choice adapts to the prospective utility of exploration, while novel stimuli persistently draw favor as a result of inflated reward expectations biasing an exploitative strategy. Crucially, novelty is proposed to actively modulate uncertainty processing, effectively blunting the influence of uncertainty in shaping the subjective utility ascribed to novel stimuli. Both behavioral data and fMRI activity sampled from the ventromedial prefrontal cortex, frontopolar cortex and ventral striatum validate this model, thereby establishing a computational account that can not only explain behavior but also shed light on the functional contribution of these key brain regions to the exploration/exploitation trade-off. Our results point to multiple strategies and neural substrates charged with balancing the explore/exploit dilemma, with each targeting distinct aspects of the decision problem to foster a manageable decomposition of an otherwise intractable task.


Item Type:Report or Paper (Discussion Paper)
Related URLs:
URLURL TypeDescription
https://doi.org/10.1101/2021.10.13.464279DOIDiscussion Paper
ORCID:
AuthorORCID
Cockburn, Jeffrey0000-0003-2004-8653
Man, Vincent0000-0001-5380-5521
Cunningham, William0000-0002-7063-5859
O'Doherty, John P.0000-0003-0016-3531
Additional Information:The copyright holder for this preprint is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under a CC-BY-NC-ND 4.0 International license. This work was funded by grants from the US National Science Foundation (1207573) to JOD, the Natural Science and Engineering Research Council (RGPIN-2018-05946) to WC, and the Social Sciences and Humanities Research Council (CGS-MSFSS) to VM. Author Contributions: J.C and J.P.O conceived of the experiments and data analysis. J.C, J.P.O, V.M, and W.C wrote the manuscript. J.C carried out the fMRI experiment, and V.M carried out the replication study. The authors have declared no competing interest.
Group:Tianqiao and Chrissy Chen Institute for Neuroscience
Funders:
Funding AgencyGrant Number
NSFSES-1207573
Natural Sciences and Engineering Research Council of Canada (NSERC)RGPIN-2018-05946
Social Sciences and Humanities Research Council (SSHRC)CGS-MSFSS
Subject Keywords:Explore; Exploit; Reinforcement learning; fMRI
DOI:10.1101/2021.10.13.464279
Record Number:CaltechAUTHORS:20211015-222205042
Persistent URL:https://resolver.caltech.edu/CaltechAUTHORS:20211015-222205042
Official Citation:Novelty and uncertainty interact to regulate the balance between exploration and exploitation in the human brain. Jeffrey Cockburn, Vincent Man, William A Cunningham, John P O'Doherty bioRxiv 2021.10.13.464279; doi: https://doi.org/10.1101/2021.10.13.464279
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:111483
Collection:CaltechAUTHORS
Deposited By: George Porter
Deposited On:18 Oct 2021 17:10
Last Modified:18 Oct 2021 17:10

Repository Staff Only: item control page