FP3: A 3D Foundation Policy for Robotic Manipulation
Rujia Yang, Geng Chen, Chuan Wen, Yang Gao
AI summary
Problem
Current robot foundation models rely exclusively on 2D images, lacking the 3D geometric reasoning needed to generalize to novel objects and scenes, and typically require hundreds of task-specific demonstrations to learn new skills.
Approach
FP3 is a 1.3B parameter diffusion transformer pre-trained on 60,000 3D point cloud trajectories from the DROID dataset, followed by parameter-efficient LoRA fine-tuning on a small set of high-quality demonstrations.
Key results
- >90% in-domain and >80% in-the-wild success rates with only 80 demonstrations
- Zero-shot generalization to unseen objects, backgrounds, and lighting
- Matches frontier VLA baselines in simulation with significantly less data
- Robust real-world task execution in 2 hours of single-GPU fine-tuning
Why it matters
Enables scalable, data-efficient robotic manipulation by proving that 3D geometric priors drastically improve sample efficiency and real-world generalization over 2D vision-language models.
Abstract
Following its success in natural language pro- cessing and computer vision, foundation models that are pre- trained on large-scale multi-task datasets have also shown great potential in robotics. However, most existing robot foundation models rely solely on 2D image observations, ignoring 3D geometric information, which is essential for robots to perceive and reason about the 3D world. In this paper, we introduce FP3, a large-scale 3D foundation policy model for robotic manipulation. FP3 builds on a scalable diffusion transformer architecture and is pre-trained on 60k trajectories with point cloud observations. With the model design and diverse pre- ∗Equal contribution. †Equal advising. Corresponding authors. training data, FP3 can be efficiently fine-tuned for down- stream tasks while exhibiting strong generalization capabilities. Experiments on real robots demonstrate that with only 80 demonstrations, FP3 is able to learn a new task with over 90% success rates in novel environments with unseen objects, significantly surpassing existing robot foundation models.