Switchable Neural Teleoperation
Jianglong Ye, Changwei Jing, Kezhou Chen, Keyi Wang, Sha Yi, Xueyan Zou, Xiaolong Wang
AI summary
Problem
Collecting high-quality teleoperation data for complex dexterous manipulation is slow and error-prone due to morphology gaps, control latency, and limited operator feedback.
Approach
The system interleaves human teleoperation with an autonomous grasping policy, using a learned neural switcher to automatically decide when to delegate control, and supports both VR hand-tracking and lightweight 6-DoF controllers.
Key results
- Increases task success rates by up to 33% across six real-world manipulation tasks
- Reduces demonstration collection time by over 70% compared to state-of-the-art teleoperation baselines
- Enables precise dexterous control using only a lightweight 6-DoF controller without finger tracking
- Achieves reliable, smooth task transitions with a learned neural switcher that matches manual switching performance
Why it matters
Provides a scalable, accessible framework for efficiently collecting high-fidelity manipulation data, accelerating robotics research and deployment.
Abstract
Collecting demonstrations through human teleop- eration is an effective approach for learning complex manip- ulation skills. However, challenges such as morphology gaps, control latency, and limited feedback make high-quality data collection costly and inefficient. In this paper, we introduce Neural Teleoperation, a shared-autonomy system that inte- grates human guidance with a robust grasping policy using a learning-based policy switcher. This hybrid framework allows users to focus on high-level planning while delegating fine- grained control to an autonomous policy when needed. Our system supports both immersive VR devices and lightweight 6-DoF controllers, making dexterous hand teleoperation more accessible. Real-world experiments across six manipulation tasks show that Neural Teleop increases success rates and reduces demonstration collection time compared to state-of- the-art baselines.