RANGER: A Monocular Zero-Shot Semantic Navigation Framework through Visual Contextual Adaptation
Ming-Ming Yu, Yi Chen, Börje F. Karlsson, Wenjun Wu
AI summary
Problem
Existing zero-shot navigation methods rely heavily on depth sensors, precise pose estimation, and global mapping, restricting real-world deployment, while lacking the ability to leverage visual in-context learning from short environmental videos for rapid adaptation.
Approach
The framework builds a dynamic keyframe-based memory bank to perform online monocular 3D reconstruction and semantic point cloud fusion, using vision-language models to guide adaptive waypoint selection and navigation without architectural modifications or retraining.
Key results
- Competitive navigation success rate and exploration efficiency on the HM3D benchmark without depth or pose inputs
- Superior visual in-context learning adaptability when provided with a short offline traversal video
- Successful real-world deployment on a humanoid robot in office and meeting room environments
- Elimination of global 3D mapping, precise camera calibration, and task-specific fine-tuning requirements
Why it matters
Enables low-cost, rapidly deployable embodied agents for real-world service and monitoring tasks by leveraging freely available monocular videos for immediate environmental adaptation.
Abstract
Efficient target localization and autonomous navi- gation in complex environments are fundamental to real-world embodied applications. While recent advances in multimodal foundation models have enabled zero-shot object goal naviga- tion, allowing robots to search for arbitrary objects without fine-tuning, existing methods face two key limitations: (1) heavy reliance on ground-truth depth and pose information, which restricts applicability in real-world scenarios; and (2) lack of visual in-context learning (VICL) capability to extract geometric and semantic priors from environmental context, as in a short traversal video. To address these challenges, we propose RANGER, a novel zero-shot, open-vocabulary semantic navigation framework that operates using only a monocular camera. Leveraging powerful 3D foundation models, RANGER eliminates the dependency on depth and pose while exhibiting strong VICL capability. By simply observing a short video of the target environment, the system can also significantly improve task efficiency without requiring architectural modifications or task-specific retraining. The framework integrates several key components: keyframe-based 3D reconstruction, semantic point cloud generation, vision-language model (VLM)-driven exploration value estimation, high-level adaptive waypoint se- lection, and low-level action execution. Experiments on the HM3D benchmark and real-world environments demonstrate that RANGER achieves competitive performance in terms of navigation success rate and exploration efficiency, while showing superior VICL adaptability, with no previous 3D mapping of the environment required.