Rapid Adaptation of Particle Dynamics for Generalized Deformable Object Mobile Manipulation
Bohan Wu, Roberto MartÃn-MartÃn, Li Fei-Fei
AI summary
Problem
Manipulating deformable objects requires adapting to unknown material properties and shape changes, but existing methods struggle with real-time adaptation, object occlusions, and zero-shot generalization to new objects or robots.
Approach
RAPID uses a two-phase simulation-to-real reinforcement learning pipeline that first conditions a visuomotor policy on privileged particle dynamics, then trains lightweight adaptation modules to infer those dynamics from real-time depth images and robot actions for direct real-world deployment.
Key results
- 80%+ success rates on real-robot 1D Inserting and 2D Covering tasks
- 65%+ performance gain over state-of-the-art methods using only onboard visual inputs
- Zero-shot generalization to unseen object dynamics, categories, instances, and lighting conditions
- Online adaptation to soft-to-stiff deformable objects without real-time state estimation
Why it matters
It provides a scalable, model-free framework for real-world mobile manipulation of unpredictable non-rigid objects, bridging the gap between simulation training and robust real-world deployment.
Abstract
We address the challenge of learning to manipu- late deformable objects with unknown dynamics. In non-rigid objects, the dynamics parameters define how they react to interactions –how they stretch, bend, compress, and move– and they are critical to determining the optimal actions to perform a manipulation task successfully. In other robotic domains, such as legged locomotion and in-hand rigid object manipulation, state-of-the-art approaches can handle unknown dynamics using Rapid Motor Adaptation (RMA). Through a supervised procedure in simulation that encodes each rigid object’s dy- namics, such as mass and position, these approaches learn a policy that conditions actions on a vector of latent dynamic parameters inferred from sequences of state-actions. However, in deformable object manipulation, the object’s dynamics not only include its mass and position, but also how the shape of the object changes. Our key insight is that the recent ground-truth particle positions of a deformable object in simulation capture changes in the object’s shape, making it possible to extend RMA to deformable object manipulation. This key insight allows us to develop RAPID, a two-phase method that learns to perform real-robot deformable object mobile manipulation by: 1) learning a visuomotor policy conditioned on the object’s dynamics embedding, which is encoded from the object’s priv- ileged information in simulation, and 2) learning to infer this embedding using non-privileged information instead, such as robot visual observations and actions, so that the learned policy can transfer to the real world. On a 22-DOF robot, RAPID enables 80%+ success rates across two real-world vision-based deformable object mobile manipulation tasks, under unseen object dynamics, categories, and instances. More details are at https://sites.google.com/view/rapid-robotics. * Equal Advising. Authors are with Stanford University and the University of Texas at Austin, USA. {bohanwu, feifeili}@stanford.edu, robertomm@utexas.edu