Ventura: Adapting Image Diffusion Models for Unified Task Conditioned Navigation
Arthur Zhang, Xiangyun Meng, Luca Callari, Dong Ki Kim, Shayegan Omidshafiei, Joydeep Biswas, Ali-akbar Agha-mohammadi, Amirreza Shaban
AI summary
Problem
Existing vision-language models struggle to translate diverse, semantically rich human instructions into precise and safe robot navigation plans due to action space mismatches and limited spatial reasoning.
Approach
VENTURA adapts a pre-trained image diffusion model to generate a task-conditioned path mask in image space, which a lightweight behavior-cloning policy then converts into executable robot waypoints.
Key results
- 33% higher success rate and 54% fewer collisions than SOTA baselines
- Emergent generalization to unseen task combinations
- Scalable auto-labeling pipeline using off-the-shelf point tracking
- Open-source language-captioned navigation dataset
Why it matters
Provides a scalable, interpretable pathway for grounding open-world language priors into precise robot control, accelerating real-world deployment of adaptive autonomous systems.
Abstract
Robots must adapt to diverse human instruc- tions and operate safely in unstructured, open-world environ- ments. Recent Vision–Language models (VLMs) offer strong priors for grounding language and perception, but remain difficult to steer for navigation due to differences in action spaces and pretraining objectives that hamper transferabil- ity to robotics tasks. Towards addressing this, we introduce VENTURA, a vision–language navigation system that finetunes internet-pretrained image diffusion models for path planning. Instead of directly predicting low-level actions, VENTURA generates a path mask (i.e. a visual plan) in image space that captures fine-grained, context-aware navigation behaviors. A lightweight behavior-cloning policy grounds these visual plans into executable trajectories, yielding an interface that follows natural language instructions to generate diverse robot behaviors. To scale training, we supervise on path masks derived from self-supervised tracking models paired with VLM- augmented captions, avoiding manual pixel-level annotation or highly engineered data collection setups. In extensive real-world evaluations, VENTURA outperforms state-of-the-art foundation model baselines on object reaching, obstacle avoidance, and terrain preference tasks, improving success rates by 33% and reducing collisions by 54% across both seen and unseen scenarios. Notably, we find that VENTURA generalizes to unseen *{ arthurz, joydeepb } @cs.utexas.edu †lcallari@uw.edu ‡{xiangyun, amir, dongki, shy, ali}@fieldai.com combinations of distinct tasks, revealing emergent compo- sitional capabilities. Videos, code, and additional materials: https://venturapath.github.io.