Real-to-Sim Robot Policy Evaluation with Gaussian Splatting Simulation of Soft-Body Interactions
Kaifeng Zhang, Shuo Sha, Hanxiao Jiang, Matthew Loper, Hyunjong Song, Guangyan Cai, Zhuo Xu, Hu Xiaochen, Changxi Zheng, Yunzhu Li
AI summary
Problem
Direct real-world evaluation of robotic manipulation policies is costly, slow, and hard to reproduce, while existing simulators fail to capture the coupled visual and physical complexity of soft-body interactions.
Approach
The framework reconstructs soft-body digital twins from interaction videos and builds a photorealistic simulator using 3D Gaussian Splatting with automated color and positional alignment, exposing it via a standard Gym interface.
Key results
- Strong sim-to-real success rate correlation across policies and tasks
- Rendering and dynamics fidelity both critical for reliable evaluation
- Multi-GPU parallelized evaluation runs several times faster than real-world trials
- Outperforms IsaacLab baseline in sim-to-real correlation for deformable tasks
Why it matters
Provides robotics researchers a scalable, reproducible, and trustworthy proxy for policy benchmarking, accelerating the iteration cycle for foundation models in robotics.
Abstract
Robotic manipulation policies are advancing rapidly, but their direct evaluation in the real world remains costly, time-consuming, and difficult to reproduce, particularly for tasks involving deformable objects. Simulation provides a scalable and systematic alternative, yet existing simulators often fail to capture the coupled visual and physical complexity of soft-body interactions. We present a real-to-sim policy evalu- ation framework that constructs soft-body digital twins from real-world videos and renders robots, objects, and environments with photorealistic fidelity using 3D Gaussian Splatting. We validate our approach on representative deformable manipula- tion tasks, including plush toy packing, rope routing, and T- block pushing, demonstrating that simulated rollouts correlate strongly with real-world execution performance and reveal key behavioral patterns of learned policies. Our results suggest that combining physics-informed reconstruction with high- quality rendering enables reproducible, scalable, and accurate evaluation of robotic manipulation policies. Website: https: //real2sim-eval.github.io/