Real-Is-Sim: Bridging the Sim-To-Real Gap with a Dynamic Digital Twin
Jad Abou-Chakra, Lingfeng Sun, Krishan Rana, Brandon May, Karl Schmeckpeper, Niko Sünderhauf, Maria Vittoria Minniti, Laura Herlant
AI summary
Problem
Translating robotic policies from simulation to reality is hindered by distribution mismatches and dynamics discrepancies, making safe pre-deployment testing and reliable transfer difficult.
Approach
The framework uses a dynamic digital twin powered by Embodied Gaussians that continuously corrects its state using real-world RGB feedback, allowing the policy to always control a simulated robot that the physical robot mirrors in real-time.
Key results
- Virtual evaluations reliably rank training checkpoints by real-world success rate
- Augmenting real demonstrations with virtual rollouts improves task performance
- Simulator-derived representations enable flexible policy conditioning
- Parallelized virtual deployment achieves 5x to 8x speedup
Why it matters
It enables roboticists to unify training, evaluation, and deployment in a single scalable pipeline, drastically reducing the time and risk of sim-to-real transfer.
Abstract
We introduce real-is-sim, a new approach to inte- grating simulation into behavior cloning pipelines. In contrast to real-only methods, which lack the ability to safely test policies before deployment, and sim-to-real methods, which require complex adaptation to cross the sim-to-real gap, our framework allows policies to seamlessly switch between running on real hardware and running in parallelized virtual environments. At the center of real-is-sim is a dynamic digital twin, powered by the Embodied Gaussian simulator, that synchronizes with the real world at 60Hz. This twin acts as a mediator between the behavior cloning policy and the real robot. Policies are trained using representations derived from simulator states and always act on the simulated robot, never the real one. During deploy- ment, the real robot simply follows the simulated robot’s joint states, and the simulation is continuously corrected with real world measurements. This setup, where the simulator drives all policy execution and maintains real-time synchronization with the physical world, shifts the responsibility of crossing the sim- to-real gap to the digital twin’s synchronization mechanisms, instead of the policy itself. We demonstrate real-is-sim on a long-horizon manipulation task (PushT), showing that virtual evaluations are consistent with real-world results. We further show how real-world data can be augmented with virtual rollouts and compare to policies trained on different repre- sentations derived from the simulator state including object poses and rendered images from both static and robot-mounted cameras. Our results highlight the flexibility of the real-is-sim framework across training, evaluation, and deployment stages. Videos available at https://real-is-sim.github.io.