Mazumdar, Eric and Dong, Roy and Rúbies Royo, Vicenç and Tomlin, Claire and Sastry, S. Shankar (2017) A Multi-Armed Bandit Approach for Online Expert Selection in Markov Decision Processes. . (Unpublished) https://resolver.caltech.edu/CaltechAUTHORS:20210903-213649911
![]() |
PDF
- Submitted Version
See Usage Policy. 562kB |
Use this Persistent URL to link to this item: https://resolver.caltech.edu/CaltechAUTHORS:20210903-213649911
Abstract
We formulate a multi-armed bandit (MAB) approach to choosing expert policies online in Markov decision processes (MDPs). Given a set of expert policies trained on a state and action space, the goal is to maximize the cumulative reward of our agent. The hope is to quickly find the best expert in our set. The MAB formulation allows us to quantify the performance of an algorithm in terms of the regret incurred from not choosing the best expert from the beginning. We first develop the theoretical framework for MABs in MDPs, and then present a basic regret decomposition identity. We then adapt the classical Upper Confidence Bounds algorithm to the problem of choosing experts in MDPs and prove that the expected regret grows at worst at a logarithmic rate. Lastly, we validate the theory on a small MDP.
Item Type: | Report or Paper (Discussion Paper) | ||||||||
---|---|---|---|---|---|---|---|---|---|
Related URLs: |
| ||||||||
ORCID: |
| ||||||||
Record Number: | CaltechAUTHORS:20210903-213649911 | ||||||||
Persistent URL: | https://resolver.caltech.edu/CaltechAUTHORS:20210903-213649911 | ||||||||
Usage Policy: | No commercial reproduction, distribution, display or performance rights in this work are provided. | ||||||||
ID Code: | 110718 | ||||||||
Collection: | CaltechAUTHORS | ||||||||
Deposited By: | George Porter | ||||||||
Deposited On: | 07 Sep 2021 16:44 | ||||||||
Last Modified: | 07 Sep 2021 16:44 |
Repository Staff Only: item control page