3D Dynamics-Aware Manipulation: Endowing Manipulation Policies with 3D Foresight
Yuxin He, Ruihao Zhang, Xianzu Wu, Zhiyuan Zhang, Cheng Ding, Qiang Nie
AI summary
Problem
Existing manipulation policies rely on 2D visual dynamics, which lack the depth information necessary for robust control in tasks involving significant forward or backward movement.
Approach
A unified framework that trains a causal transformer to simultaneously predict current depth, future RGB-D frames, and 3D flow from monocular inputs, explicitly endowing the policy with 3D foresight.
Key results
- State-of-the-art success rates on CALVIN and LIBERO simulation benchmarks
- Significant real-world performance gains on depth-critical tasks like stacking and drawer manipulation
- Negligible inference latency increase (~6 ms) by offloading auxiliary prediction heads during deployment
- Complementary performance benefits from jointly predicting current depth, future RGB-D, and 3D flow
Why it matters
Enables robotics developers to build more robust, depth-aware manipulation policies efficiently without compromising real-time control requirements.
Abstract
The incorporation of world modeling into manipu- lation policy learning has pushed the boundary of manipulation performance. However, existing efforts simply model the 2D visual dynamics, which is insufficient for robust manipulation when target tasks involve prominent depth-wise movement. To address this, we present a 3D dynamics-aware manipulation framework that seamlessly integrates 3D world modeling and policy learning. Three self-supervised learning tasks (current depth estimation, future RGB-D prediction, 3D flow prediction) are introduced within our framework, which complement each other and endow the policy model with 3D foresight. Extensive experiments on simulation and the real world show that 3D foresight can greatly boost the performance of manipulation policies without sacrificing inference speed. Code is available at https://github.com/Stardust-hyx/3D-Foresight.