Learning to Play Table Tennis from Scratch Using Muscular Robots

Dynamic tasks like table tennis are relatively easy to learn for humans, but pose significant challenges to robots. Such tasks require accurate control of fast movements and precise tim- ing in the presence of imprecise state estimation of the flying ball and the robot. Reinforcement learning (RL) has shown promise in learning complex control tasks from data. However, applying step-based RL to dynamic tasks on real systems is safety-critical as RL requires exploring and failing safely for millions of time steps in high-speed and high-acceleration regimes. This paper demonstrates that using robot arms driven by pneumatic artificial muscles (PAMs) enables safe end-to-end learning of table tennis using model-free RL. In particular, we learn from scratch for thousands of trials while a stochastic policy acts on the low- level controls of the real system. The robot returns and smashes real balls with 5 m s−1 and 12 m s−1 on average respectively to a desired landing point. Additionally, we present HYSR, a practical hybrid sim and real training procedure that avoids training with real balls by virtually replaying recorded ball trajectories and applying actions to the real robot. To the best of our knowledge, this work pioneers (a) fail-safe learning of a safety-critical dynamic task using anthropomorphic robot arms, (b) learning a precision-demanding problem with a PAM-driven system that is inherently hard to control as well as (c) train a robot to play table tennis without real balls. Videos, code and datasets can be found on muscularTT.embodied.ml.