GaussTwin: Unified Simulation and Correction with Gaussian Splatting for Robotic Digital Twins
Yichen Cai, Paul Jansonnie, Cristiana de Farias, Oleg Arenz, Jan Peters
AI summary
Problem
Existing digital twins lack a unified physics model capable of handling both rigid and deformable objects while bridging the real-to-simulation gap, which limits downstream applications like model predictive control.
Approach
The method combines Position-Based Dynamics with a discrete Cosserat rod model for physically grounded simulation, then anchors 3D Gaussians to these physical primitives and refines them using photometric error and segmentation masks for stable visual correction.
Key results
- Unified PBD simulation for rigid bodies and deformable linear objects
- Coherent Gaussian motion constrained by physical primitives to prevent drift
- Improved tracking accuracy and robustness over shape-matching baselines
- Successful downstream push-based planning on a Franka Research 3 robot
Why it matters
Enables reliable, real-time digital twins for closed-loop robotic control and policy learning by bridging the real-to-simulation gap for complex object interactions.
Abstract
Digital twins promise to enhance robotic manip- ulation by maintaining a consistent link between real-world perception and simulation. However, most existing systems struggle with the lack of a unified model, complex dynamic interactions, and the real-to-sim gap, which limits downstream applications such as model predictive control. Thus, we propose GaussTwin, a real-time digital twin that combines position- based dynamics with discrete Cosserat rod formulations for physically grounded simulation, and Gaussian splatting for efficient rendering and visual correction. By anchoring Gaus- sians to physical primitives and enforcing coherent SE(3) updates driven by photometric error and segmentation masks, GaussTwin achieves stable prediction-correction while preserv- ing physical fidelity. Through experiments in both simulation and on a Franka Research 3 platform, we show that GaussT- win consistently improves tracking accuracy and robustness compared to shape-matching and rigid-only baselines, while also enabling downstream tasks such as push-based planning. These results highlight GaussTwin as a step toward unified, physically meaningful digital twins that can support closed-loop robotic interaction and learning. Code and videos are available at https://6cyc6.github.io/gstwin/.