Re$^3$Sim: Generating High-Fidelity Simulation Data Via 3D-Photorealistic Real-To-Sim for Robotic Manipulation
Xiaoshen Han, Junqiu Yu, Minghuan Liu, Yilun Chen, Xiaoyang Lyu, Yang Tian, Bolun Wang, Weinan Zhang, Jiangmiao Pang
AI summary
Problem
Real-world data collection for robotics is costly and time-consuming, while existing simulations suffer from significant visual and geometric sim-to-real gaps that hinder effective policy transfer.
Approach
The system bridges these gaps by separately reconstructing scene geometry for collision detection and using 3D Gaussian Splatting for photorealistic rendering, then aligning the simulated environment with the real world to generate high-fidelity expert data for imitation learning.
Key results
- Achieves zero-shot sim-to-real transfer with >58% average success rate using only ~10 minutes of simulated data collection.
- Delivers high-fidelity hybrid rendering that minimizes visual and geometric sim-to-real gaps.
- Enables rapid scene reconstruction in under three minutes with minimal human effort.
- Demonstrates that doubling synthetic dataset size significantly boosts imitation learning performance to match real-world data.
Why it matters
Provides a scalable, low-cost pipeline for generating high-quality simulation data, reducing reliance on expensive real-world data collection and accelerating the development of generalizable robotic manipulation policies.
Abstract
Real-world data collection for robotics is costly and resource-intensive, requiring skilled operators and expensive hardware. Simulations offer a scalable alternative but often fail to achieve sim-to-real generalization due to geometric and visual gaps. To address these challenges, we propose a 3D-photorealistic real-to-sim system, namely, RE3SIM, addressing geometric and visual sim-to-real gaps. RE3SIM employs advanced 3D reconstruction and rendering techniques to faithfully recreate real-world scenarios, enabling real-time rendering of simulated cross-view cameras within a physics-based simulator. By utilizing privileged information to collect expert demonstrations efficiently in simulation, and train robot policies with imitation learning, we validate the effectiveness of the real-to-sim-to-real system across various manipulation task scenarios. Notably, with only simulated data, we can achieve zero-shot sim-to-real transfer with an average success rate exceeding 58%. To push the limit of real-to-sim, we further generate a large-scale simulation dataset, demonstrating how a robust policy can be built from simulation data that generalizes across various objects.