AugVLA-3D: Depth-Driven Feature Augmentation for Vision-Language-Action Models
Zhifeng Rao,∗, Wenlong Chen∗, Lei Xie, Xia Hua, Dongfu Yin, Zhen Tian, F. Richard Yu
AI summary
Problem
Existing Vision-Language-Action models rely on 2D visual features, lacking explicit 3D structural awareness needed for robust spatial reasoning and manipulation. Prior 3D-enhanced approaches require expensive specialized sensors or large-scale 3D datasets, limiting their scalability and practical deployment.
Approach
The framework converts standard RGB images into dense 3D point clouds using monocular depth estimation, extracts geometric descriptors with PointNet, and aligns them with control objectives via a lightweight Action Assistant regularizer.
Key results
- Sensor-free 3D feature extraction from RGB images using VGGT depth estimation
- Lightweight Action Assistant module for task-aligned feature regularization
- Superior action prediction accuracy and robustness in geometrically ambiguous scenarios
- Scalable training compatibility with existing large-scale 2D VLA datasets
Why it matters
Provides a scalable, hardware-efficient pathway to enhance robotic spatial reasoning and manipulation without relying on costly 3D sensors or datasets.
Abstract
Vision-Language-Action (VLA) models have re- cently achieved remarkable progress in robotic perception and control, yet most existing approaches primarily rely on VLM trained using 2D images, which limits their spatial under- standing and action grounding in complex 3D environments. To address this limitation, we propose a novel framework that integrates depth estimation into VLA models to enrich 3D feature representations. Specifically, we employ a depth estimation baseline called VGGT to extract geometry-aware 3D cues from standard RGB inputs, enabling efficient utilization of existing large-scale 2D datasets while implicitly recovering 3D structural information. To further enhance the reliability of these depth-derived features, we introduce a new module called action assistant, which constrains the learned 3D rep- resentations with action priors and ensures their consistency with downstream control tasks. By fusing the enhanced 3D features with conventional 2D visual tokens, our approach significantly improves the generalization ability and robustness of VLA models. Experimental results demonstrate that the proposed method not only strengthens perception in geomet- rically ambiguous scenarios but also leads to superior action prediction accuracy. This work highlights the potential of depth- driven data augmentation and auxiliary expert supervision for bridging the gap between 2D observations and 3D-aware decision-making in robotic systems.