Gotta Scoop 'Em All: Sim-And-Real Co-Training of Graph-Based Neural Dynamics for Long-Horizon Scooping
Kaiwen Hong, Haonan Chen, Jiaming Xu, Runxuan Wang, Kaylan Wang, Mingtong Zhang, Shuijing Liu, Yifan Zhu, Yunzhu Li, Katherine Driggs-Campbell
AI summary
Problem
Scooping all granular objects from a container requires long-horizon planning and accurate modeling of complex particle-scoop-container interactions, which pure simulation or limited real-world data cannot achieve alone.
Approach
The system co-trains a graph-based neural dynamics model using large-scale synthetic data and a small amount of real-world data, then employs Monte Carlo Tree Search with behavior primitives to plan and execute long-horizon scooping sequences.
Key results
- Co-trained dynamics model significantly reduces prediction error over sim-only and real-only baselines
- Achieves 100% success rate in fewer than 20 scoops across multiple materials and containers
- Zero-shot generalization to new granular materials and rapid adaptation to new container shapes
- Introduction of the SAGO task with a complete sim-to-real planning and execution pipeline
Why it matters
Provides a scalable, data-efficient framework for complex granular manipulation, advancing applications in healthcare, industrial assembly, and scientific exploration.
Abstract
We present a system to address a new problem of removing all granular objects from a container using a scoop, a task that requires long-horizon reasoning over the complex interactions between granular objects, the scoop, and the container. To tackle this challenge, we adopt a graph- based neural dynamics (GBND) model trained through sim- and-real co-training that leverages both rich synthetic data and a small amount of real-world data. Using the learned model, we plan long-horizon scooping sequences of behavior primitives with Monte Carlo Tree Search (MCTS). Extensive experimental evaluations demonstrate that our system with the co-training strategy is capable of removing all materials in a container with fewer than 20 scoops, significantly outperforming strategies that use only simulation or real data. Our approach can also generalize in a zero-shot manner to new materials, and quickly adapt to new containers with few additional data points. More videos and analyses are available on our project website https://scoopthemall.github.io/.