DexTele: A Dual-Arm Dexterous Teleoperation System Based on Motion Retargeting and Adaptive Force Control
yuanchuan lai, Qing Gao, Ziyan Liang, Xianfeng Cheng, Junjie Hu, Zhaojie Ju
AI summary
Problem
Existing teleoperation systems lack cross-platform motion retargeting generalization and struggle with adaptive grasping for diverse objects, often relying on platform-specific paired data or fixed force thresholds.
Approach
The system maps human motions to robots using a vision-based SAG-GCN encoder-decoder for cross-platform retargeting, while an adaptive grasping module uses a vision-language model to estimate target forces and model predictive control to optimize joint commands in real time.
Key results
- Precise cross-platform motion retargeting across RMC-DA, YuMi, and Unitree H1 robots
- Adaptive grasping of diverse objects via VLM force estimation and MPC optimization
- Dual-stream SAG-GCN architecture enabling accurate arm-hand motion mapping without paired datasets
- Validated generalization across multiple robot architectures and unseen objects in simulations and real-world tests
Why it matters
Provides a unified, scalable teleoperation framework that improves reliability and safety for remote manipulation across heterogeneous robotic platforms.
Abstract
In dual-arm dexterous teleoperation, cross- platform generalization of motion retargeting and interactivity of grasping are crucial. However, the heterogeneity of robotic architectures and the wide variety of grasping objects pose significant challenges to achieving precise motion retargeting and compliant grasping in dual-arm dexterous teleoperation. To address these challenges, a dual-arm dexterous teleoperation system (DexTele) is proposed based on motion retargeting and adaptive force control. First, a vision-based motion retargeting module is designed to generate preliminary robot motions from human images. In this module, a motion-graph encoder and la- tent optimization are proposed for precise and convenient cross- platform motion retargeting. Second, an adaptive grasping module is designed to achieve compliant grasping. This module combines a vision-language model (VLM) with model predictive control (MPC), allowing the system to predict the required grasping force for a target object and perform gradient-based online optimization. Finally, extensive experiments demonstrate that the DexTele achieves precise motion retargeting and compliant grasping with generalization across multiple robot platforms. Project can be found at: https://github.io/DexTele.