← Back ICRA 2026

Unlocking the Potential of Soft Actor-Critic for Imitation Learning

Nayari Marie Lessa, Melya Boukheddimi, Frank Kirchner

PDF

AI summary

Key figure (auto-extracted from paper)

Integrating the off-policy Soft Actor-Critic algorithm with Adversarial Motion Priors yields higher imitation rewards and more robust quadruped locomotion than the standard on-policy PPO baseline.

Imitation Learning Soft Actor-Critic Adversarial Motion Priors Quadruped Locomotion Bio-inspired Robotics Reinforcement Learning

Problem

Current imitation learning frameworks for robotics predominantly rely on Proximal Policy Optimization (PPO), which prioritizes stability over sample efficiency and policy generalization, limiting the generation of natural, adaptable bio-inspired motions.

Approach

The authors combine Adversarial Motion Priors (AMP) with the off-policy Soft Actor-Critic (SAC) algorithm, using replay buffers and entropy regularization to guide a quadruped robot in imitating animal gaits across diverse terrains.

Key results

Achieves higher AMP discriminator rewards compared to the AMP+PPO baseline
Accurately reproduces dog-inspired walk and trot gaits with matching joint ranges of motion
Maintains stable locomotion and smooth gait transitions across flat and undulating terrains
Demonstrates superior sample efficiency and policy generalization over on-policy methods

Why it matters

Offers a more sample-efficient and robust alternative to standard on-policy methods for training bio-inspired robotic locomotion, advancing adaptive quadruped robot development.

Abstract

Learning-based methods have enabled robots to acquire bio-inspired movements with increasing levels of natu- ralness and adaptability. Among these, Imitation Learning (IL) has proven effective in transferring complex motion patterns from animals to robotic systems. However, current state-of- the-art frameworks predominantly rely on Proximal Policy Optimization (PPO), an on-policy algorithm that prioritizes stability over sample efficiency and policy generalization. This paper proposes a novel IL framework that combines Adversarial Motion Priors (AMP) with the off-policy Soft Actor-Critic (SAC) algorithm to overcome these limitations. This integration leverage replay-driven learning and entropy- regularized exploration, enabling naturalistic behavior and task execution improving data efficiency and robustness. We evaluate the proposed approach (AMP+SAC) on quadruped gaits involving multiple reference motions and diverse terrains. Experimental results demonstrate that the proposed framework not only maintains stable task execution but also achieves higher imitation rewards compared to the widely used AMP+PPO method. These findings highlight the potential of an off-policy IL formulations for advancing motion generation in robotics. Code and supplementary material are available at: https: //github.com/nayariml/AMP_SAC.git

Index terms

Imitation Learning Bioinspired Robot Learning Reinforcement Learning