A Caltech Library Service

Distributed Reinforcement Learning in Multi-Agent Networked Systems

Lin, Yiheng and Qu, Guannan and Huang, Longbo and Wierman, Adam (2020) Distributed Reinforcement Learning in Multi-Agent Networked Systems. . (Unpublished)

[img] PDF - Submitted Version
See Usage Policy.


Use this Persistent URL to link to this item:


We study distributed reinforcement learning (RL) for a network of agents. The objective is to find localized policies that maximize the (discounted) global reward. In general, scalability is a challenge in this setting because the size of the global state/action space can be exponential in the number of agents. Scalable algorithms are only known in cases where dependencies are local, e.g., between neighbors. In this work, we propose a Scalable Actor Critic framework that applies in settings where the dependencies are non-local and provide a finite-time error bound that shows how the convergence rate depends on the depth of the dependencies in the network. Additionally, as a byproduct of our analysis, we obtain novel finite-time convergence results for a general stochastic approximation scheme and for temporal difference learning with state aggregation that apply beyond the setting of RL in networked systems.

Item Type:Report or Paper (Discussion Paper)
Related URLs:
URLURL TypeDescription Paper
Lin, Yiheng0000-0001-6524-2877
Qu, Guannan0000-0002-5466-3550
Huang, Longbo0000-0002-7341-447X
Wierman, Adam0000-0002-5923-0199
Additional Information:We see no ethical concerns related to this paper.
Record Number:CaltechAUTHORS:20201014-143549786
Persistent URL:
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:106067
Deposited By: Tony Diaz
Deposited On:14 Oct 2020 21:43
Last Modified:02 Jun 2023 01:06

Repository Staff Only: item control page