FiLM-Nav: Efficient and Generalizable Navigation Via VLM Fine-Tuning
Naoki Yokoyama, Sehoon Ha
AI summary
Problem
Foundation models possess strong semantic knowledge but lack the spatial-temporal grounding needed for efficient embodied decision-making, often relying on complex intermediate representations or zero-shot prompting that limit real-world applicability.
Approach
FiLM-Nav directly fine-tunes a pre-trained VLM to act as a navigation policy that selects the next best exploration frontier by conditioning on raw egocentric visual history and a language goal, trained on a diverse mix of navigation tasks and an auxiliary spatial reasoning objective.
Key results
- State-of-the-art SPL and success rate on HM3D ObjectNav among open-vocabulary methods
- State-of-the-art SPL on the HM3D-OVON benchmark
- Strong generalization to unseen object categories and synonyms
- Demonstrates that diverse task mixture and auxiliary spatial training are critical for robust navigation
Why it matters
Offers a streamlined, effective pathway for adapting large foundation models to real-world robotic navigation without relying on complex intermediate maps or separate planning modules.
Abstract
Enabling robotic assistants to navigate complex environments and locate objects described in free-form language is critical for real-world deployment. While foundation models, particularly Vision-Language Models (VLMs), offer powerful semantic understanding, effectively adapting their web-scale knowledge for embodied decision-making remains a key chal- lenge. We present FiLM-Nav (Fine-tuned Language Model for Navigation), an approach that directly fine-tunes a pre-trained VLM as the navigation policy. In contrast to methods that use foundation models primarily in a zero-shot manner or for map annotation, FiLM-Nav learns to select the next best exploration frontier by conditioning directly on raw visual trajectory history and the navigation goal. Leveraging targeted simulated embodied experience allows the VLM to ground its powerful pre-trained representations in the specific dynamics and visual patterns relevant to goal-driven navigation. Critically, fine-tuning on a diverse data mixture combining ObjectNav, OVON, ImageNav, and an auxiliary spatial reasoning task proves essential for achieving robustness and broad generalization. FiLM-Nav sets a new state-of-the-art in both SPL and success rate on HM3D ObjectNav among open-vocabulary methods, and sets a state- of-the-art SPL on the challenging HM3D-OVON benchmark, demonstrating strong generalization to unseen object categories. Our work validates that directly fine-tuning VLMs on diverse simulated embodied data is a highly effective pathway towards generalizable and efficient semantic navigation capabilities. 1NY and SH are with Georgia Institute of Technology nyokoyama@gatech.edu