A Caltech Library Service

Trust Region Policy Optimization for POMDPs

Azizzadenesheli, Kamyar and Bera, Manish Kumar and Anandkumar, Animashree (2018) Trust Region Policy Optimization for POMDPs. . (Unpublished)

[img] PDF - Submitted Version
See Usage Policy.


Use this Persistent URL to link to this item:


We propose Generalized Trust Region Policy Optimization (GTRPO), a policy gradient Reinforcement Learning (RL) algorithm for both Markov decision processes (MDP) and Partially Observable Markov Decision Processes (POMDP). Policy gradient is a class of model-free RL methods. Previous policy gradient methods are guaranteed to converge only when the underlying model is an MDP and the policy is run for an infinite horizon. We relax these assumptions to episodic settings and to partially observable models with memory-less policies. For the latter class, GTRPO uses a variant of the Q-function with only three consecutive observations for each policy updates, and hence, is computationally efficient. We theoretically show that the policy updates in GTRPO monotonically improve the expected cumulative return and hence, GTRPO has convergence guarantees.

Item Type:Report or Paper (Discussion Paper)
Related URLs:
URLURL TypeDescription Paper
Azizzadenesheli, Kamyar0000-0001-8507-1868
Anandkumar, Animashree0000-0002-6974-6797
Additional Information:K. Azizzadenesheli is supported in part by NSF Career Award CCF-1254106 and Air Force FA9550-15-1-0221. A. Anandkumar is supported in part by Microsoft Faculty Fellowship, Google Faculty Research Award, Adobe Grant, NSF Career Award CCF-1254106, and AFOSR YIP FA9550-15-1-0221.
Funding AgencyGrant Number
Air Force Office of Scientific Research (AFOSR)FA9550-15-1-0221
Microsoft Faculty FellowshipUNSPECIFIED
Google Faculty Research AwardUNSPECIFIED
Record Number:CaltechAUTHORS:20190327-085807408
Persistent URL:
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:94179
Deposited By: George Porter
Deposited On:28 Mar 2019 14:32
Last Modified:23 Dec 2022 19:20

Repository Staff Only: item control page