CaltechAUTHORS
  A Caltech Library Service

Duality between density function and value function with applications in constrained optimal control and Markov Decision Process

Chen, Yuxiao and Ames, Aaron D. (2019) Duality between density function and value function with applications in constrained optimal control and Markov Decision Process. . (Unpublished) https://resolver.caltech.edu/CaltechAUTHORS:20190410-120558713

[img] PDF - Submitted Version
See Usage Policy.

1058Kb

Use this Persistent URL to link to this item: https://resolver.caltech.edu/CaltechAUTHORS:20190410-120558713

Abstract

Density function describes the density of states in the state space with some initial state distribution. Its evolution follows the Liouville Partial Differential Equation (PDE). We show that the density function is the dual of the value function in the optimal control problems and strong duality holds. By utilizing the duality, constraints that are hard to enforce in the primal value function optimization such as safety constraints in robot navigation, traffic capacity constraints in traffic flow control can be posed on the density function, and the constrained optimal control problem can be solved with a primal-dual algorithm that alternates between the primal and dual optimization. The primal optimization follows the standard optimal control algorithm with a perturbation term generated by the density constraint, and the dual problem solves the Liouville PDE to get the density function under a fixed control strategy and updates the perturbation term. We show examples in robot navigation and traffic control to demonstrate the capability of the proposed formulation.


Item Type:Report or Paper (Discussion Paper)
Related URLs:
URLURL TypeDescription
http://arxiv.org/abs/1902.09583arXivDiscussion Paper
ORCID:
AuthorORCID
Ames, Aaron D.0000-0003-0848-3177
Record Number:CaltechAUTHORS:20190410-120558713
Persistent URL:https://resolver.caltech.edu/CaltechAUTHORS:20190410-120558713
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:94623
Collection:CaltechAUTHORS
Deposited By: George Porter
Deposited On:11 Apr 2019 18:20
Last Modified:03 Oct 2019 21:05

Repository Staff Only: item control page