A Caltech Library Service

Hierarchical Imitation and Reinforcement Learning

Le, Hoang M. and Jiang, Nan and Agarwal, Alekh and Dudík, Miroslav and Yue, Yisong and Daumé, Hal, III (2018) Hierarchical Imitation and Reinforcement Learning. Proceedings of Machine Learning Research, 80 . pp. 2917-2926. ISSN 1938-7228.

[img] PDF - Published Version
See Usage Policy.

[img] PDF - Submitted Version
See Usage Policy.

[img] PDF - Supplemental Material
See Usage Policy.


Use this Persistent URL to link to this item:


We study how to effectively leverage expert feedback to learn sequential decision-making policies. We focus on problems with sparse rewards and long time horizons, which typically pose significant challenges in reinforcement learning. We propose an algorithmic framework, called hierarchical guidance, that leverages the hierarchical structure of the underlying problem to integrate different modes of expert interaction. Our framework can incorporate different combinations of imitation learning (IL) and reinforcement learning (RL) at different levels, leading to dramatic reductions in both expert effort and cost of exploration. Using long-horizon benchmarks, including Montezuma’s Revenge, we demonstrate that our approach can learn significantly faster than hierarchical RL, and be significantly more label-efficient than standard IL. We also theoretically analyze labeling cost for certain instantiations of our framework.

Item Type:Article
Related URLs:
URLURL TypeDescription
Yue, Yisong0000-0001-9127-1989
Additional Information:© 2018 by the author(s). The majority of this work was done while HML was an intern at Microsoft Research. HML is also supported in part by an Amazon AI Fellowship.
Funding AgencyGrant Number
Microsoft ResearchUNSPECIFIED
Amazon Web ServicesUNSPECIFIED
Record Number:CaltechAUTHORS:20190205-113025214
Persistent URL:
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:92671
Deposited By: Tony Diaz
Deposited On:05 Feb 2019 19:37
Last Modified:05 Oct 2022 16:21

Repository Staff Only: item control page