ViTac-Tracing: Visual-Tactile Imitation Learning of Deformable Object Tracing
Yongqiang Zhao, Haining Luo, Yupeng Wang, Emmanouil Spyrakos-Papastavridis, Yiannis Demiris, SHAN LUO
AI summary
Problem
Existing tracing methods either lack generalizability across different deformable object categories or fail in real-world settings due to reliance on precise object modeling and sim-to-real transfer gaps.
Approach
The authors train a unified transformer-based policy using demonstrations from a custom visual-tactile teleoperation system, augmented by a local center loss for contact stability and a global task loss for progress tracking.
Key results
- 80% average success rate on seen 1D and 2D deformable objects
- 65% success rate on unseen objects, demonstrating strong generalization
- Budget-efficient teleoperation system with real-time visual-tactile streaming and singularity vibration feedback
- Ablation studies confirm the individual contributions of local center and global task losses
Why it matters
Provides a scalable, model-free solution for real-world deformable object manipulation, benefiting applications like cable routing, cloth handling, and assistive dressing.
Abstract
Deformable objects often appear in unstructured configurations. Tracing deformable objects helps bringing them into extended states and facilitating the downstream manipula- tion tasks. Due to the requirements for object-specific modeling or sim-to-real transfer, existing tracing methods either lack gen- eralizability across different categories of deformable objects or struggle to complete tasks reliably in the real world. To address this, we propose a novel visual-tactile imitation learning method to achieve one-dimensional (1D) and two-dimensional (2D) deformable object tracing with a unified model. Our method is designed from both local and global perspectives based on visual and tactile sensing. Locally, we introduce a weighted loss that emphasizes actions maintaining contact near the center of the tactile image, improving fine-grained adjustment. Globally, we propose a tracing task loss that helps the policy to regulate task progression. On the hardware side, to compensate for the limited features extracted from visual information, we integrate tactile sensing into a low-cost teleoperation system considering both the teleoperator and the robot. Extensive ablation and comparative experiments on diverse 1D and 2D deformable objects demonstrate the effectiveness of our approach, achieving an average success rate of 80% on seen objects and 65% on unseen objects. Demos, code and datasets are available at https://sites.google.com/view/vitac-tracing.