Few-Shot Neural Differentiable Simulator: Real-To-Sim Rigid-Contact Modeling
Zhenhao Huang, Siyuan Luo, Bingyang Zhou, Ziqiu Zeng, Fan Shi
AI summary
Problem
Analytical simulators struggle to capture complex real-world contact dynamics and are sensitive to hard-to-measure parameters, while learning-based simulators require massive, costly real-world datasets. This creates a trade-off between physical accuracy, computational efficiency, and data scalability.
Approach
The method identifies contact parameters from just a few real-world trajectories, scales them into diverse synthetic data, and trains a mesh-based graph neural network with derived surrogate gradients to achieve fully differentiable collision detection and rigid-body dynamics.
Key results
- Contact parameter identification reduces MuJoCo trajectory error from 1.14 to 0.73 using only three real trajectories.
- The GNN simulator matches or slightly outperforms differentiable baseline Brax and identified MuJoCo in replicating real-world trajectories.
- Surrogate gradients enable end-to-end differentiability through discrete collision detection, supporting gradient-based optimization.
- Simulation-based policy learning in multi-object scenarios demonstrates improved efficiency and simulation fidelity with minimal supervision.
Why it matters
Enables researchers and roboticists to train accurate, differentiable simulators for contact-rich manipulation tasks using minimal real-world data, accelerating policy learning and control development.
Abstract
Accurate physics simulation is essential for robotic learning and control, yet analytical simulators often fail to cap- ture complex contact dynamics, while learning-based simulators typically require large amounts of costly real-world data. To bridge this gap, we propose a few-shot real-to-sim approach that combines the physical consistency of analytical formulations with the representational capacity of graph neural network (GNN)- based models. Using only a small amount of real-world data, our method calibrates analytical simulators to generate large- scale synthetic datasets that capture diverse contact interac- tions. On this foundation, we introduce a mesh-based GNN that implicitly models rigid-body forward dynamics and derive surrogate gradients for collision detection, achieving full differ- entiability. Experimental results demonstrate that our approach enables learning-based simulators to outperform differentiable baselines in replicating real-world trajectories. In addition, the differentiable design supports gradient-based optimization, which we validate through simulation-based policy learning in multi- object interaction scenarios. Extensive experiments show that our framework not only improves simulation fidelity with minimal supervision but also increases the efficiency of policy learning. Taken together, these findings suggest that differentiable simu- lation with few-shot real-world grounding provides a powerful direction for advancing future robotic manipulation and control.