Mixed Reality-Based, Immersive, Semi-Autonomous Robotic Telemanipulation for the Execution of Peg-In-Hole Tasks
Shifei Duan, Francesco De Pace, Zhe Wang, Minas Liarokapis
AI summary
Problem
Existing semi-autonomous telemanipulation systems rely on complex calibration, specialized hardware, and lack intuitive interfaces, making precise robotic assembly difficult to deploy and cognitively demanding for operators.
Approach
The system uses a consumer MR headset for natural hand-gesture teleoperation combined with a computer vision alignment estimator and an adaptive velocity control strategy that dynamically slows robot motion as the peg approaches the hole.
Key results
- Stabilizes end-effector motion by eliminating oscillations and overshooting during fine alignment
- Reduces trajectory length compared to standard telemanipulation without adaptive control
- Achieves reliable task execution using only coarse calibration and standard CPU processing
- Lowers operator cognitive workload while maintaining high precision in semi-autonomous mode
Why it matters
This accessible, low-cost MR framework enables non-experts to perform precise robotic assembly tasks efficiently without relying on expensive sensors or complex autonomous algorithms.
Abstract
Semi-autonomy in telemanipulation frameworks has the potential to reduce user cognitive load while preserving human perceptual oversight and decision-making capabilities. However, existing semi-autonomous telemanipulation systems are heavily dependent on calibration and hardware configu- rations, making rapid deployment difficult. Moreover, existing VR-based telemanipulation systems lack intuitive interaction mechanisms, requiring users to manage complex control inter- faces. To address these limitations, we introduce an intuitive and immersive semi-autonomous robotic telemanipulation system that leverages a mixed reality (MR) headset with minimal hardware requirements. Requiring only CPU processing and coarse calibration procedures, the system combines human perception with autonomous control strategies through nat- ural hand tracking and finger gestures to achieve precise, reliable task execution. To validate this approach, we conducted thorough evaluations involving complex peg-in-hole tasks and compared performance with and without the proposed control strategy. The results highlight that our system demonstrates robust performance, and the proposed control strategy further enhances its stability and effectiveness.