SoftMimicGen: A Data Generation System for Scalable Robot Learning in Deformable Object Manipulation
Masoud Moghani, Mahdi Azizian, Animesh Garg, Yuke Zhu, Sean Huver, Ajay Uday Mandlekar
AI summary
Problem
Collecting large-scale robot datasets is costly and labor-intensive, while existing synthetic data methods are limited to rigid-body tasks, leaving a critical gap for deformable object manipulation essential to real-world applications.
Approach
The pipeline starts with a small set of human teleoperated demonstrations and automatically generates large-scale datasets for novel deformable objects by using non-rigid registration to adaptively warp trajectories to new object states and contexts.
Key results
- Released a high-fidelity simulation suite for diverse deformable objects and tasks across four robot platforms
- Generated thousands of demonstrations per task and trained high-performing visuomotor policies via imitation learning
- Achieved zero-shot sim-to-real transfer on real-world hardware with further gains from sim-real co-training
- Demonstrated broad compatibility with varied robot embodiments and complex manipulation behaviors
Why it matters
It bridges a major gap in robotics by enabling scalable synthetic data generation for deformable manipulation, accelerating robot foundation model development and drastically reducing reliance on costly real-world data collection.
Abstract
Large-scale robot datasets have facilitated the learning of a wide range of robot manipulation skills, but these datasets remain difficult to collect and scale further, owing to the intractable amount of human time, effort, and cost required. Simulation and synthetic data generation have proven to be an effective alternative to fuel this need for data, especially with the advent of recent work showing that such synthetic datasets can dramatically reduce real- world data requirements and facilitate generalization to novel scenarios unseen in real-world demonstrations. However, this paradigm has been limited to rigid-body tasks, which are easy to simulate. Deformable object manipulation encompasses a large portion of real-world manipulation and remains a crucial gap to address towards increasing adoption of the synthetic simulation data paradigm. In this paper, we introduce SOFTMIMICGEN, an automated data generation pipeline for deformable object manipulation tasks. We introduce a suite of high-fidelity simulation environments that encompasses a wide range of deformable objects (stuffed animal, rope, tissue, towel) and manipulation behaviors (high-precision threading, dynamic whipping, folding, pick-and-place), across four robot embodiments: a single-arm manipulator, bimanual arms, a humanoid, and a surgical robot. We apply SOFTMIMICGEN to generate datasets across the task suite, train high-performing policies from the data, and systematically analyze the data generation system. Project website: softmimicgen.github.io.