CaltechAUTHORS
  A Caltech Library Service

Task complexity interacts with state-space uncertainty in the arbitration process between model-based and model-free reinforcement-learning at both behavioral and neural levels

Kim, Dongjae and Park, Geon Yeong and O'Doherty, John P. and Lee, Sang Wan (2018) Task complexity interacts with state-space uncertainty in the arbitration process between model-based and model-free reinforcement-learning at both behavioral and neural levels. . (Unpublished) http://resolver.caltech.edu/CaltechAUTHORS:20180927-114223662

[img] PDF - Submitted Version
See Usage Policy.

1355Kb

Use this Persistent URL to link to this item: http://resolver.caltech.edu/CaltechAUTHORS:20180927-114223662

Abstract

A major open question concerns how the brain governs the allocation of control between two distinct strategies for learning from reinforcement: model-based and model-free reinforcement learning. While there is evidence to suggest that the reliability of the predictions of the two systems is a key variable responsible for the arbitration process, another key variable has remained relatively unexplored: the role of task complexity. By using a combination of novel task design, computational modeling, and model-based fMRI analysis, we examined the role of task complexity alongside state-space uncertainty in the arbitration process between model-based and model-free RL. We found evidence to suggest that task complexity plays a role in influencing the arbitration process alongside state-space uncertainty. Participants tended to increase model-based RL control in response to increasing task complexity. However, they resorted to model-free RL when both uncertainty and task complexity were high, suggesting that these two variables interact during the arbitration process. Computational fMRI revealed that task complexity interacts with neural representations of the reliability of the two systems in the inferior prefrontal cortex bilaterally. These findings provide insight into how the inferior prefrontal cortex negotiates the trade-off between model-based and model-free RL in the presence of uncertainty and complexity, and more generally, illustrates how the brain resolves uncertainty and complexity in dynamically changing environments.


Item Type:Report or Paper (Discussion Paper)
Related URLs:
URLURL TypeDescription
https://doi.org/10.1101/393983DOIDiscussion Paper
ORCID:
AuthorORCID
Kim, Dongjae0000-0002-4513-9087
Lee, Sang Wan0000-0001-6266-9613
Additional Information:The copyright holder for this preprint is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. We thank Peter Dayan for insightful comments and Ralph Lee for his assistance. This work was funded by grants R01DA033077 and R01DA040011 to J.P.O.D. from the National Institute on Drug Abuse. This work was also supported by an Institute for Information & Communications Technology Promotion (IITP) grant funded by the Korea government (No. 2017-0-00451), the ICT R&D program of MSIP/IITP. [2016-0-00563, Research on Adaptive Machine Learning Technology Development for Intelligent Autonomous Digital Companion], the Brain Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science, ICT & Future Planning (NRF-2016M3C7A1914448), the National Research Foundation of Korea (NRF) grant funded by the Korea government(MSIT) (No. 2017R1C1B2008972), the research fund of the Korea Advanced Institute of Science and Technology (KAIST) under Grant code G04150045, and Samsung Research Funding Center of Samsung Electronics under Project Number SRFC-TC1603-06. Author contributions: SL and JOD conceived and designed the study. SL implemented the behavioral task and ran the fMRI study. DK, GP, and SL designed computational models and analyzed the data. SL, JOD, and DK wrote the paper. All authors approved the final version for submission.
Funders:
Funding AgencyGrant Number
NIHR01DA033077
NIHR01DA040011
Institute for Information & Communications Technology Promotion2017-0-00451
Institute for Information & Communications Technology Promotion2016-0-00563
National Research Foundation of KoreaNRF-2016M3C7A1914448
National Research Foundation of Korea2017R1C1B2008972
Korea Advanced Institute of Science and Technology (KAIST)G04150045
Samsung ElectronicsSRFC-TC1603-06
Record Number:CaltechAUTHORS:20180927-114223662
Persistent URL:http://resolver.caltech.edu/CaltechAUTHORS:20180927-114223662
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:89999
Collection:CaltechAUTHORS
Deposited By: George Porter
Deposited On:28 Sep 2018 14:23
Last Modified:28 Sep 2018 14:23

Repository Staff Only: item control page