VO-DP: Semantic-Geometric Adaptive Diffusion Policy for Vision-Only Robotic Manipulation
Zehao Ni, Yonghao He, Lingfeng Qian, Jilei Mao, Fa Fu, Wei Sui, Hu Su, Junran Peng, Zhipeng Wang, Bin He
AI summary
Problem
Vision-only visuomotor policies currently lack robust representation learning, failing to match the accuracy of point-cloud-based methods, while 3D-aware approaches depend on expensive, complex hardware.
Approach
VO-DP extracts semantic features from DINOv2 and geometric features from the VGGT model, fuses them via cross-attention, compresses them with a CNN, and conditions a diffusion policy head on the combined representation.
Key results
- 64.6% average success rate in simulation, matching point-cloud baseline DP3
- 87.9% real-world success rate, significantly outperforming DP3 and DP baselines
- High robustness to lighting, background, and object size variations
- Open-sourced multi-GPU training framework compatible with RoboTwin
Why it matters
Demonstrates that low-cost RGB cameras can achieve high-accuracy, robust manipulation, lowering hardware barriers for practical robotic deployment.
Abstract
In the context of imitation learning, visuomotor- based diffusion policy learning is one of the main directions in robotic manipulation. Most of these approaches rely on point clouds as observation inputs and construct scene represen- tations through point clouds feature learning, which enables them to achieve remarkable accuracy. However, the existing literature lacks an in-depth exploration of vision-only solutions that have significant potential. In this paper, we propose a Vision-Only and single-view Diffusion Policy learning method (VO-DP) that leverages pretrained visual foundation models to achieve effective fusion of semantic and geometric features. We utilize intermediate features from VGGT incorporating semantic features from DINOv2 and geometric features from Alternating Attention blocks. Features are fused via cross- attention and spatially compressed with a CNN to form the input to the policy head. Extensive experiments demonstrate that VO-DP not only outperforms the vision-only baseline DP significantly but also exhibits distinct performance trends against the point cloud-based method DP3: in simulation tasks, VO-DP achieves an average success rate of 64.6%—on par with DP3 64.0% and far higher than DP 34.8%, while in real-world tasks, it reaches 87.9%, outperforming both DP3 67.5% and DP 11.2% by a notable margin. Further robust- ness evaluations confirm that VO-DP remains highly stable under varying conditions including color, size, background, and lighting. Lastly, we open-source a robotic manipulation training library: it supports multi-machine/multi-GPU parallel training and mixed precision, is compatible with visuomotor policies (DP, DP3) and the RoboTwin simulator, and will be released upon publication.