ViTacGen: Robotic Pushing with Vision-To-Touch Generation
Zhiyuan Wu, Yijiong Lin, Yongqiang Zhao, Xuyang Zhang, Zhuo Chen, Nathan Lepora, SHAN LUO
AI summary
Problem
Real tactile sensors are costly, fragile, and difficult to calibrate, while vision-only policies struggle to capture the subtle contact dynamics necessary for precise robotic pushing.
Approach
The framework uses an encoder-decoder network (VT-Gen) to generate synthetic tactile depth images from visual sequences, which are then fused with visual data via contrastive learning in a reinforcement learning policy (VT-Con).
Key results
- Achieved a success rate of up to 86%
- Outperformed baseline methods in simulation and real-world experiments
- Enabled effective zero-shot deployment on visual-only robotic systems
- Standardized tactile representation using contact depth maps to overcome sensor manufacturing variations
Why it matters
It eliminates the reliance on high-resolution physical tactile sensors while maintaining the performance benefits of touch-based manipulation.
Abstract
Robotic pushing is a fundamental manipulation task that requires tactile feedback to capture subtle contact forces and dynamics between the end-effector and the object. However, real tactile sensors often face hardware limitations such as high costs and fragility, and deployment challenges involving calibration and variations between different sensors, while vision-only poli- cies struggle with satisfactory performance. Inspired by humans’ ability to infer tactile states from vision, we propose ViTacGen, a novel robot manipulation framework designed for visual robotic pushing with vision-to-touch generation in reinforcement learning to eliminate the reliance on high-resolution real tactile sensors, enabling effective zero-shot deployment on visual-only robotic systems. Specifically, ViTacGen consists of an encoder-decoder vision-to-touch generation network that generates contact depth images, a standardized tactile representation, directly from visual image sequence, followed by a reinforcement learning policy that fuses visual-tactile data with contrastive learning based on visual and generated tactile observations. We validate the effectiveness of our approach in both simulation and real world experiments, demonstrating its superior performance and achieving a success rate of up to 86%. Code and data are available on https://robot- perception-lab.github.io/vitacgen-website/.