Manipulating Elasto-Plastic Objects with 3D Occupancy and Learning-Based Predictive Control
Zhen ZHANG, Xiangyu CHU, Yunxi TANG, Lulu ZHAO, Jing HUANG, Zhongliang Jiang, K. W. Samuel Au
AI summary
Problem
Manipulating elasto-plastic volumetric objects is hindered by severe self-occlusion, complex high-dimensional dynamics, and irreversible deformation, making accurate state representation and control extremely difficult.
Approach
The method infers dense 3D occupancy from multi-view RGB images, learns deformation dynamics with a hybrid 3D CNN and GNN, and uses model predictive control guided by a shape-based action initialization to plan manipulation sequences.
Key results
- Novel 3D occupancy prediction network for dense volumetric state inference
- Hybrid 3D CNN and GNN dynamics model capturing complex elasto-plastic transitions
- Shape-based action initialization module improving MPC planning efficiency
- Validated successful goal-shaping in both simulation and real-world experiments
Why it matters
Provides a robust, data-driven pipeline for precise robotic manipulation of complex deformable materials, advancing applications in healthcare, manufacturing, and domestic robotics.
Abstract
Manipulating elasto-plastic objects remains a sig- nificant challenge due to severe self-occlusion, difficulties of representation, and complicated dynamics. This work proposes a novel framework for elasto-plastic object manipulation with a quasi-static assumption for motions, leveraging 3D occupancy to represent such objects, a learned dynamics model trained with 3D occupancy, and a learning-based predictive control algorithm to address these challenges effectively. We build a novel data collection platform to collect full spatial information and propose a pipeline for generating a 3D occupancy dataset. To infer the 3D occupancy during manipulation, an occupancy prediction network is trained with multiple RGB images supervised by the generated dataset. We design a deep neural network empow- ered by a 3D convolution neural network (CNN) and a graph neural network (GNN) to predict the complex deformation with the inferred 3D occupancy results. A learning-based predictive control algorithm is introduced to plan the robot’s actions, incorporating a novel shape-based action initialization module specifically designed to improve the planner’s efficiency. The proposed framework in this paper can successfully shape the elasto-plastic objects into a given goal shape and has been verified in various experiments both in simulation and the real world.