RL-Augmented Adaptive Model Predictive Control for Bipedal Locomotion Over Challenging Terrain
Junnosuke Kamohara, Feiyang Wu, Chinmayee Wamorkar, Seth Hutchinson, Ye Zhao
AI summary
Problem
Standard MPC struggles with terrain interaction modeling and lacks adaptivity on rough ground, while RL lacks explicit constraint guarantees and requires extensive reward shaping. Existing hybrid approaches remain largely limited to flat terrain or quadrupedal robots.
Approach
The authors employ a hierarchical architecture where an RL policy learns residual parameters to adapt the MPC’s dynamics model, swing leg trajectory, and gait frequency, preserving optimization-based constraint satisfaction while adding reactive adaptability.
Key results
- RL policy learns residual dynamics, swing trajectory, and gait frequency adaptations
- Significantly improved robustness on stairs, stepping stones, and low-friction surfaces
- Enables reactive recovery from foot entrapment and severe slippage
- Ablation studies validate each learned residual module's contribution
Why it matters
Bridges the gap between model-based safety and learning-based adaptability, offering a practical control framework for humanoids navigating complex real-world environments.
Abstract
Model predictive control (MPC) has demonstrated effectiveness for humanoid bipedal locomotion; however, its applicability in challenging environments, such as rough and slippery terrain, is limited by the difficulty of modeling terrain interactions. In contrast, reinforcement learning (RL) has achieved notable success in training robust locomotion policies over diverse terrain, yet it lacks guarantees of constraint satisfaction and often requires substantial reward shaping. Recent efforts in combining MPC and RL have shown promise of taking the best of both worlds, but they are primarily restricted to flat terrain or quadrupedal robots. In this work, we propose an RL-augmented MPC framework tailored for bipedal locomotion over rough and slippery terrain. Our method parametrizes three key components of single- rigid-body-dynamics-based MPC: system dynamics, swing leg controller, and gait frequency. We validate our approach through bipedal robot simulations in NVIDIA IsaacLab across various terrains, including stairs, stepping stones, and low- friction surfaces. Experimental results demonstrate that our RL-augmented MPC framework produces significantly more adaptive and robust behaviors compared to baseline MPC and RL. Project page: https://rl-augmented-mpc.github. io/rlaugmentedmpc/