Rainbow-DemoRL: Combining Improvements in Demonstration-Augmented Reinforcement Learning
Dwait Bhatt, Shih-Chieh Chou, Nikolay Atanasov
AI summary
Problem
Online reinforcement learning struggles with low sample efficiency and unsafe exploration, making it costly for real-world robotics. Despite numerous demonstration-augmented methods, it remains unclear which components are most effective and how they can be optimally combined.
Approach
The authors classify demonstration-augmented RL into three strategies, conduct a large-scale empirical study to isolate each component's contribution, and systematically evaluate hybrid combinations for sample-efficient online learning.
Key results
- Direct offline data reuse and behavior cloning initialization consistently outperform complex offline RL pretraining
- Hybrid combinations of buffer prefilling and simple BC initialization yield the highest sample efficiency gains
- Action mixing strategies show high performance variance and are less reliable than direct data or initialization methods
- Provides a comprehensive benchmark and practical guidelines for combining demonstration-augmented RL components
Why it matters
It provides robotics researchers and practitioners with clear, evidence-based guidelines for efficiently accelerating online RL training on physical hardware using offline demonstrations.
Abstract
Several approaches have been proposed to im- prove the sample efficiency of online reinforcement learning (RL) by leveraging demonstrations collected offline. The offline data can be used directly as transitions to optimize RL objectives, or offline policy and value functions can first be learned from the data and then used for online finetuning or to provide reference actions. While each of these strategies has shown compelling results, it is unclear which method has the most impact on sample efficiency, whether these approaches can be combined, and if there are cumulative benefits. We classify existing demonstration-augmented RL approaches into three categories and perform an extensive empirical study of their strengths, weaknesses, and combinations to isolate the contribution of each strategy and determine effective hybrid combinations for sample-efficient online RL. Our analysis reveals that directly reusing offline data and initializing with behavior cloning consistently outperform more complex offline RL pretraining methods for improving online sample efficiency.