Using Non-Expert Data to Robustify Imitation Learning Via Offline Reinforcement Learning
Kevin Huang, Rosario Scalise, Cleah Winston, Ayush Agrawal, Yunchu Zhang, Rohan Baijal, Markus Grotz, Byron Boots, Benjamin Burchfiel, Hongkai Dai, Masha Itkina, Paarth Shah, Abhishek Gupta
AI summary
Problem
Standard imitation learning relies on expensive expert demonstrations and struggles to recover from out-of-distribution states, while conventional offline RL fails to effectively utilize abundant but suboptimal non-expert data due to sparse coverage and overly conservative policy extraction.
Approach
RISE assigns binary rewards to expert versus non-expert data to train an offline RL critic that stitches recovery trajectories back to expert states, overcoming sparse coverage by enforcing policy Lipschitz continuity and widening the action distribution via spectral norm penalties and data augmentation.
Key results
- Introduces RISE, an offline RL framework for robustifying imitation learning with non-expert data
- Identifies offline RL stitching pitfalls in low-coverage regimes and proposes spectral norm regularization and data augmentation
- Demonstrates effective use of diverse non-expert data types including play, suboptimal demonstrations, and policy rollouts
- Validates significant recovery and generalization improvements across simulation manipulation and real-world furniture assembly tasks
Why it matters
Enables roboticists to train robust, generalizable manipulation policies using cheap, abundant, and imperfect real-world data without expensive expert demonstrations or complex reward engineering.
Abstract
Imitation learning has proven effective for training robots to perform complex tasks from expert human demonstra- tions. However, it remains limited by its reliance on high-quality, task-specific data, restricting adaptability to the diverse range of real-world object configurations and scenarios. In contrast, non-expert data—such as play data, suboptimal demonstrations, partial task completions, or rollouts from suboptimal policies— can offer broader coverage and lower collection costs. However, conventional imitation learning approaches fail to utilize this data effectively. To address these challenges, we posit that with right design decisions, offline reinforcement learning can be used as a tool to harness non-expert data to enhance the performance of imitation learning policies. We show that while standard offline RL approaches can be ineffective at actually leveraging non- expert data under the sparse data coverage settings typically en- countered in the real world, simple algorithmic modifications can allow for the utilization of this data, without significant additional assumptions. Our approach shows that broadening the support of the policy distribution can allow imitation algorithms augmented by offline RL to solve tasks robustly, showing considerably enhanced recovery and generalization behavior. In manipulation tasks, these innovations significantly increase the range of initial conditions where learned policies are successful when non-expert data is incorporated. Moreover, we show that these methods are able to leverage all collected data, including partial or suboptimal demonstrations, to bolster task-directed policy performance. This underscores the importance of algorithmic techniques for using non-expert data for robust policy learning in robotics.