ViTaS: Visual Tactile Soft Fusion Contrastive Learning for Visuomotor Learning
Yufeng Tian, Shuiqi Cheng, Tianming Wei, Tianxing Zhou, Yuanhang Zhang, Zixian Liu, Qianwei Han, Zhecheng Yuan, Huazhe Xu
AI summary
Problem
Existing visuo-tactile fusion methods rely on direct concatenation or simple patching, failing to exploit the inherent alignment and complementarity between modalities, which limits performance in occluded or complex manipulation scenarios.
Approach
ViTaS introduces soft fusion contrastive learning to align visual and tactile embeddings in latent space, combined with a conditional variational autoencoder (CVAE) to reconstruct visual observations from fused features, thereby leveraging cross-modal complementarity.
Key results
- State-of-the-art performance across 12 simulated and 3 real-world tasks
- Superior generalization to varied object geometries and randomized targets
- Robust performance under significant tactile/visual noise and self-occlusion
- Effective integration with Diffusion Policy to compensate for limited visual input
Why it matters
Provides a practical, robust framework for robotic manipulation that leverages touch to overcome visual limitations, advancing real-world deployment of dexterous robots.
Abstract
Tactile information plays a crucial role in human manipulation tasks and has recently garnered increasing at- tention in robotic manipulation. However, existing approaches mostly focus on the alignment of visual and tactile features and the integration mechanism tends to be direct concatenation. Consequently, they struggle to effectively cope with occluded scenarios due to neglecting the inherent complementary nature of both modalities and the alignment may not be exploited enough, limiting the potential of their real-world deployment. In this paper, we present ViTaS, a simple yet effective framework that incorporates both visual and tactile information to guide the behavior of an agent. We introduce Soft Fusion Contrastive Learning, an advanced version of conventional contrastive learning method and a CVAE module to utilize the alignment and complementarity within visuo-tactile representations. We demonstrate the effectiveness of our method in 12 simulated and 3 real-world environments, and our experiments show that ViTaS significantly outperforms existing baselines. Project page: https://skyrainwind.github.io/ViTaS/index.html.