Towards Robot to Human Skill Coaching: A ML-Powered IoT and HRI Platform for Martial Arts Training

Advances in human sensing and machine learning are paving the way for new applications of robotics in sports and fitness, making skill coaching smarter, easier and more accessible. Physical and social human robot interaction in par- ticular has received special attention as a feedback mechanism for human performance augmentation. A core challenge in de- ploying robots that interact physically with humans in dynamic environments such as sports, relates to modeling human skills and designing appropriate interaction schemes. We present the first ML-based HRI platform for physical robot to human skill coaching in real-time in Martial Arts which can be extended to various sports. Our system comprises of the Sawyer robot, our specially developed IoT katana and a skill-training program for the Martial Art of Iaido. We built and deployed in real-time a ML-based Iaido strike recognition model trained on expert and beginner data, and achieved accuracies ranging between 94.8% and 99.97%. We assessed the system’s effectiveness in coaching skills through robot interaction in a sparring experiment and a survey involving 12 participants practicing key Iaido techniques with guided training from Sawyer. Our results demonstrated improvement in all participants’ Iaido strike skill after training with Sawyer, and they responded positively to robot-assisted skill coaching.