TacFlex: Multi-Mode Tactile Imprints Simulation for Visuotactile Sensors with Coating Patterns
Chaofan Zhang, Shaowei Cui, Jingyi Hu, Tianyu Jiang, Tiandong Zhang, Rui Wang, Shuo Wang
AI summary
Problem
Existing visuotactile simulators fail to accurately model elastomer deformation and diverse surface patterns while ignoring multi-medium refraction, creating a large sim-to-real gap that hinders robot learning.
Approach
The framework uses Finite Element Methods to physically simulate elastomer deformation, maps the deformed mesh to arbitrary coating patterns and 3D point clouds, and applies a fast ray-tracing-based rectification model to correct optical refraction artifacts.
Key results
- Supports multi-mode tactile imprints including arbitrary coating patterns and 3D point clouds
- Introduces a millisecond-scale ray-tracing rectification method to correct multi-medium refraction
- Achieves zero-shot Sim2Real transfer for cylindrical object pose estimation
- Enables direct deployment of peg-in-hole manipulation policies across varying real-world clearances
Why it matters
Provides an open-source, high-fidelity simulation environment that accelerates contact-rich robot learning and reduces reliance on time-consuming real-world data collection.
Abstract
Visuotactile sensors have been shown to provide rich contact information for robots. However, how to build a high- fidelity visuotactile simulator that supports multi-mode tactile imprints and various sensor configurations (such as coating patterns) remains a challenging problem. In this paper, we present TacFlex, an efficient and flexible simulator for visuotactile sensors, which physically simulates the elastomer deformation using Finite Element Methods (FEM), and focuses on linking the deformed elastomer mesh to diverse tactile imprints, including tactile images with arbitrary coating patterns and tactile 3D point clouds. We further propose a ray tracing-based rectification method to deal with multi-medium refraction effects to make the simulated tactile images more realistic. Extensive qualitative and quantitative experiments are conducted to demonstrate the effectiveness of TacFlex on several visuotactile sensors. Fur- thermore, we explore the Sim2Real performance of different tactile imprints provided by TacFlex in tactile perception and manipulation tasks, such as cylindrical object pose estimation and peg-in-hole. The perception/policy models trained in simu- lation are successfully deployed in the real world. Finally, we present the outlook on the potential of TacFlex in visuotactile manipulation learning. The TacFlex simulator is open-sourced to the community. See supplementary video, code, and results at https://sites.google.com/view/tacflex/.