A Differential Dynamic Programming Framework for Inverse Reinforcement Learning
Kun Cao, Xinhang Xu, Wanxin Jin, Karl H. Johansson, Lihua Xie
AI summary
Problem
Existing inverse reinforcement learning methods rely on open-loop imitation loss functions that incorrectly assume temporally independent noise, causing biased cost estimation when learning from closed-loop expert demonstrations. Additionally, efficiently computing gradients for constrained optimal control within IRL remains computationally challenging.
Approach
The authors repurpose Differential Dynamic Programming (DDP) to efficiently compute analytical gradients for the outer IRL optimization loop and introduce a closed-loop loss function that matches expert feedback policies instead of raw trajectories.
Key results
- Unified DDP-based IRL framework for learning cost parameters, dynamics, and constraints
- Efficient gradient computation via one-step DDP recursion on an augmented system
- Novel closed-loop loss function that outperforms traditional open-loop imitation loss
- Theoretical recoverability conditions for general constrained IRL validated in simulations and real-world quadrotor experiments
Why it matters
It enables more accurate and efficient reward function learning from real-world expert demonstrations, directly benefiting robotics, autonomous systems, and control researchers working on imitation learning.
Abstract
A differential dynamic programming (DDP)-based framework for inverse reinforcement learning (IRL) is introduced to recover the parameters in the cost function, system dynamics, and constraints from demonstrations. Different from existing work, where DDP was usually used for the inner forward problem, our proposed framework uses it to efficiently compute the gradient required in the outer inverse problem with equality and inequality constraints. The equivalence between the proposed and existing methods based on Pontryagin’s Maximum Princi- ple (PMP) is established. More importantly, using this DDP- based IRL with an open-loop loss function, a closed-loop IRL framework is presented. In this framework, a loss function is proposed to capture the closed-loop nature of demonstrations. It is shown to be better than the commonly used open-loop loss function. We show that the closed-loop IRL framework reduces to a constrained inverse optimal control problem under certain assumptions. Under these assumptions and a rank condition, it is proven that the learning parameters can be recovered from the demonstration data. The proposed framework is extensively evaluated through four numerical robot examples and one real- world quadrotor system. The experiments validate the theoretical results and illustrate the practical relevance of the approach.