CaltechAUTHORS
  A Caltech Library Service

Robust Reinforcement Learning: A Constrained Game-theoretic Approach

Yu, Jing and Gehring, Clement and Schäfer, Florian and Anandkumar, Animashree (2021) Robust Reinforcement Learning: A Constrained Game-theoretic Approach. Proceedings of Machine Learning Research, 144 . pp. 1242-1254. ISSN 1938-7228. https://resolver.caltech.edu/CaltechAUTHORS:20210727-172214672

[img] PDF - Published Version
See Usage Policy.

362kB

Use this Persistent URL to link to this item: https://resolver.caltech.edu/CaltechAUTHORS:20210727-172214672

Abstract

Deep reinforcement learning (RL) methods provide state-of-art performance in complex control tasks. However, it has been widely recognized that RL methods often fail to generalize due to unaccounted uncertainties. In this work, we propose a game theoretic framework for robust reinforcement learning that comprises many previous works as special cases. We formulate robust RL as a constrained minimax game between the RL agent and an environmental agent which represents uncertainties such as model parameter variations and adversarial disturbances. To solve the competitive optimization problems arising in our framework, we propose to use competitive mirror descent (CMD). This method accounts for the interactive nature of the game at each iteration while using Bregman divergences to adapt to the global structure of the constraint set. We demonstrate an RRL policy gradient algorithm that leverages Lagrangian duality and CMD. We empirically show that our algorithm is stable for large step sizes, resulting in faster convergence on linear quadratic games.


Item Type:Article
Related URLs:
URLURL TypeDescription
http://proceedings.mlr.press/v144/yu21a.htmlPublisherArticle
Additional Information:© 2021 J. Yu, C. Gehring, F. Schäfer & A. Anandkumar. We thank the anonymous referees for their valuable feedback. CG gratefully acknowledges support from NSF grant 1723381; from AFOSR grant FA9550-17-1-0165; from ONR grant N00014-18-1-2847 and from the MIT-IBM Watson Lab. FS gratefully acknowledges support by the Air Force Office of Scientific Research under award number FA9550-18-1-0271 (Games for Computation and Learning) and the Ronald and Maxine Linde Institute of Economic and Management Sciences at Caltech. AA is supported in part by the Bren endowed chair, Microsoft, Google, Facebook and Adobe faculty fellowships.
Funders:
Funding AgencyGrant Number
NSFIIS-1723381
Air Force Office of Scientific Research (AFOSR)FA9550-17-1-0165
Office of Naval Research (ONR)N00014-18-1-2847
Massachusetts Institute of Technology (MIT)UNSPECIFIED
Air Force Office of Scientific Research (AFOSR)FA9550-18-1-0271
Linde Institute of Economic and Management ScienceUNSPECIFIED
Bren Professor of Computing and Mathematical SciencesUNSPECIFIED
MicrosoftUNSPECIFIED
GoogleUNSPECIFIED
FacebookUNSPECIFIED
AdobeUNSPECIFIED
Subject Keywords:robust reinforcement learning, zero-sum game, adversarial training, competitive optimization, policy gradient
Record Number:CaltechAUTHORS:20210727-172214672
Persistent URL:https://resolver.caltech.edu/CaltechAUTHORS:20210727-172214672
Official Citation:Yu, J., Gehring, C., Schäfer, F., Anandkumar, A. (2021). Robust Reinforcement Learning: A Constrained Game-theoretic Approach. Proceedings of the 3rd Conference on Learning for Dynamics and Control, in Proceedings of Machine Learning Research 144:1242-1254. Available from http://proceedings.mlr.press/v144/yu21a.html.
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:110027
Collection:CaltechAUTHORS
Deposited By: Tony Diaz
Deposited On:28 Jul 2021 19:53
Last Modified:28 Jul 2021 19:53

Repository Staff Only: item control page