A Caltech Library Service

Dueling Posterior Sampling for Preference-Based Reinforcement Learning

Novoseller, Ellen R. and Wei, Yibing and Sui, Yanan and Yue, Yisong and Burdick, Joel W. (2020) Dueling Posterior Sampling for Preference-Based Reinforcement Learning. Proceedings of Machine Learning Research, 124 . pp. 1029-1038. ISSN 2640-3498.

[img] PDF - Published Version
See Usage Policy.

[img] PDF - Submitted Version
See Usage Policy.


Use this Persistent URL to link to this item:


In preference-based reinforcement learning (RL), an agent interacts with the environment while receiving preferences instead of absolute feedback. While there is increasing research activity in preference-based RL, the design of formal frameworks that admit tractable theoretical analysis remains an open challenge. Building upon ideas from preference-based bandit learning and posterior sampling in RL, we present DUELING POSTERIOR SAMPLING (DPS), which employs preference-based posterior sampling to learn both the system dynamics and the underlying utility function that governs the preference feedback. As preference feedback is provided on trajectories rather than individual state-action pairs, we develop a Bayesian approach for the credit assignment problem, translating preferences to a posterior distribution over state-action reward models. We prove an asymptotic Bayesian no-regret rate for DPS with a Bayesian linear regression credit assignment model. This is the first regret guarantee for preference-based RL to our knowledge. We also discuss possible avenues for extending the proof methodology to other credit assignment models. Finally, we evaluate the approach empirically, showing competitive performance against existing baselines.

Item Type:Article
Related URLs:
URLURL TypeDescription Paper
Novoseller, Ellen R.0000-0001-5263-0598
Sui, Yanan0000-0002-9480-627X
Yue, Yisong0000-0001-9127-1989
Additional Information:© 2021 by the author(s). This work was supported by NIH grant EB007615 and an Amazon graduate fellowship.
Funding AgencyGrant Number
Record Number:CaltechAUTHORS:20190905-154320945
Persistent URL:
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:98462
Deposited By: George Porter
Deposited On:05 Sep 2019 22:54
Last Modified:03 Jan 2022 18:59

Repository Staff Only: item control page