SignBot: Learning Human-To-Humanoid Sign Language Interaction
Guanren Qiao, Sixu Lin, Ronglai Zuo, Zhizheng Wu, Kui JIA, Guiliang Liu
AI summary
Problem
Existing sign language AI systems are largely software-based and lack physical embodiment, while current robotic approaches rely on teleoperation or lack the dexterity and stability needed for accurate gesture execution.
Approach
The framework decouples upper-body gesture tracking from lower-body balance control for robust motion execution, while integrating a cerebral module for real-time sign translation, semantic response, and gesture generation.
Key results
- Accurate sign language gesture tracking across diverse datasets and robots
- Successful sim-to-real transfer of a decoupled control policy
- Real-time bidirectional sign language translation and generation
- Generalization to distinct humanoid embodiments (legged and wheeled platforms)
Why it matters
It delivers a scalable, physically embodied solution for accessible communication, advancing both assistive technology and human-robot interaction research.
Abstract
Sign language is a natural and visual form of language that uses movements and expressions to convey meaning, serving as a crucial means of communication for individuals who are deaf or hard-of-hearing (DHH). However, the number of people proficient in sign language remains limited, highlighting the need for technological advancements to bridge communication gaps and foster interactions with minorities. Based on recent advancements in embodied humanoid robots, we propose SignBot, a novel framework for human-robot sign language interaction. SignBot integrates a cerebellum-inspired motion control component and a cerebral-oriented module for comprehension and interaction. Specifically, SignBot consists of: 1) Motion Retargeting, which converts human sign language datasets into robot-compatible kinematics; 2) Motion Control, which leverages a learning-based paradigm to develop a robust humanoid control policy for tracking sign language gestures; and 3) Generative Interaction, which incorporates translator, responder, and generator of sign language, thereby enabling natural and effective communication between robots and humans. Simulation and real-world experimental results demonstrate that SignBot can effectively facilitate human-robot interaction and perform sign language motions with diverse robots and datasets. SignBot represents a significant advancement in automatic sign language interaction on embodied humanoid robot platforms, providing a promising solution to improve communication ac- cessibility for the DHH community. Please refer to our webpage: https://qiaoguanren.github.io/SignBot-demo/