Grow Your Limits: Continuous Improvement with Real-World RL for Robotic Locomotion

Deep reinforcement learning can enable robots to autonomously acquire complex behaviors such as legged locomotion. However, RL in the real world is complicated by constraints on efficiency, safety, and overall training stability, which limits its practical applicability. We present APRL, a policy regularization framework that modulates the robot’s exploration throughout training, striking a balance between flexible improvement potential and focused, efficient explo- ration. APRL enables a quadrupedal robot to efficiently learn to walk entirely in the real world within minutes and continue to improve with more training where prior work saturates in performance. We demonstrate that continued training with APRL results in a policy that is substantially more capable of navigating challenging situations and adapts to changes in dy- namics. Videos and code to reproduce our results are available at: https://sites.google.com/berkeley.edu/aprl