Interpretable Multimodal Gesture Recognition for Drone and Mobile Robot Teleoperation Via Log-Likelihood Ratio Fusion
Seungyeol Baek, Jaspreet Singh, Lala Ray,, Hymalai Bello, Paul Lukowicz,, and Sungho Suh
AI summary
Problem
Vision-based gesture recognition for hands-free robot teleoperation degrades under real-world conditions like occlusion and poor lighting, while existing multimodal wearable approaches lack interpretability regarding how different sensors contribute to decisions.
Approach
The framework fuses inertial data from dual wristwatches and capacitive signals from custom gloves using a log-likelihood ratio strategy, which transparently quantifies each sensor's contribution to gesture classification decisions.
Key results
- Achieves recognition accuracy comparable to state-of-the-art vision-based methods
- Significantly reduces computational cost, model size, and training time
- Introduces a novel multimodal dataset of 20 aircraft marshalling-inspired gestures
- Provides interpretable modality contribution analysis via LLR values and ablation studies
Why it matters
It enables robust, real-time, and transparent hands-free teleoperation for drones and mobile robots in hazardous environments where vision-based systems fail, benefiting emergency responders and robotics engineers.
Abstract
Human operators are still frequently exposed to hazardous environments such as disaster zones and industrial facilities, where intuitive and reliable teleoperation of mobile robots and Unmanned Aerial Vehicles (UAVs) is essential. In this context, hands-free teleoperation enhances operator mobility and situational awareness, thereby improving safety in haz- ardous environments. While vision-based gesture recognition has been explored as one method for hands-free teleoperation, its performance often deteriorates under occlusions, lighting variations, and cluttered backgrounds, limiting its applicability in real-world operations. To overcome these limitations, we propose a multimodal gesture recognition framework that integrates inertial data (accelerometer, gyroscope, and orien- tation) from Apple Watches on both wrists with capacitive sensing signals from custom gloves. We design a late fusion strategy based on the log-likelihood ratio (LLR), which not only enhances recognition performance but also provides in- terpretability by quantifying modality-specific contributions. To support this research, we introduce a new dataset of 20 distinct gestures inspired by aircraft marshalling signals, comprising synchronized RGB video, IMU, and capacitive sensor data. Experimental results demonstrate that our framework achieves performance comparable to a state-of-the-art vision-based base- line while significantly reducing computational cost, model size, and training time, making it well suited for real-time robot control. We therefore underscore the potential of sensor-based multimodal fusion as a robust and interpretable solution for gesture-driven mobile robot and drone teleoperation.