OsmAG-LLM: Zero-Shot Open-Vocabulary Object Navigation Via Semantic Maps and Large Language Models Reasoning
Fujing Xie, Sören Schwertfeger, Hermann Blum
AI summary
Problem
High-fidelity object maps quickly become outdated in dynamic spaces, while existing open-vocabulary navigation methods lack scalability or require overly specific user instructions.
Approach
The system builds a compact, text-based hierarchical map that LLMs parse to predict likely object locations, then uses an online open-vocabulary detection loop to verify and retrieve targets, even if they were never initially mapped.
Key results
- Extends osmAG to object-level semantic mapping with LLM-parsable text format
- Develops online retrieval algorithm combining LLM planning with open-vocabulary detection
- Achieves higher success rates at shorter path lengths for static objects
- Significantly outperforms baselines in dynamic and unmapped object scenarios
Why it matters
Enables scalable, long-term indoor robot navigation in real-world settings where environments change and users issue natural language queries without prior map knowledge.
Abstract
Recent open-vocabulary robot mapping methods enrich dense geometric maps with pre-trained visual-language features, achieving a high level of detail and guiding robots to find objects specified by open-vocabulary language queries. While the issue of scalability for such approaches has received some attention, another fundamental problem is that high-detail object mapping quickly becomes outdated, as objects get moved around a lot. In this work, we develop a mapping and navigation system for object-goal navigation that, from the ground up, considers the possibilities that a queried object can have moved, or may not be mapped at all. Instead of striving for high-fidelity mapping detail, we consider that the main purpose of a map is to provide environment grounding and context, which we combine with the semantic priors of LLMs to reason about object locations and deploy an active, online approach to navigate to the objects. Through simulated and real-world experiments we find that our approach tends to have higher retrieval success at shorter path lengths for static objects and by far outperforms prior approaches in cases of dynamic or unmapped object queries. We provide our code and dataset at: https://github.com/xiexiexiaoxiexie/osmAG- LLM.