TOPO-Bench: An Open-Source Topological Mapping Evaluation Framework with Quantifiable Perceptual Aliasing
Jiaming Wang, Jizhuo Chen, Diwen Liu, Jiaxuan Da, Jiamo Hu, Zhiwei Xue, Linh Kästner, Harold Soh
AI summary
Problem
Topological mapping lacks standardized evaluation metrics, datasets, and protocols, hindering fair comparison. Perceptual aliasing remains a critical but poorly quantified challenge that undermines system reliability.
Approach
The paper formalizes topological consistency and proves localization accuracy as a reliable surrogate metric. It introduces a quantitative ambiguity measurement, curates a diverse benchmark dataset, and releases open-source baselines and evaluation tools.
Key results
- Formalized topological consistency metrics and localization accuracy as a surrogate measure
- Introduced the first quantitative framework for measuring dataset ambiguity
- Curated and released a diverse benchmark dataset with controlled ambiguity levels
- Demonstrated that topology-only methods struggle under perceptual aliasing, highlighting the need for additional cues
Why it matters
Enables consistent, reproducible benchmarking for robotics researchers developing robust SLAM-free topological navigation systems.
Abstract
Topological mapping offers a compact and ro- bust representation for navigation, but progress in the field is hindered by the lack of standardized evaluation metrics, datasets, and protocols. Existing systems are evaluated in different environments under different criteria, preventing fair and reproducible comparison. Moreover, a key challenge— perceptual aliasing—remains under-quantified despite its strong influence on system performance. We address these gaps by (i) formalizing topological consistency as the fundamental property of topological maps and showing that, under mild assumptions, localization accuracy provides an efficient and interpretable sur- rogate metric, and (ii) introducing the first quantitative measure of dataset ambiguity for fair comparison across environments. To support this protocol, we curate a diverse benchmark dataset with calibrated ambiguity levels, implement and release deep learning-based baseline systems, and evaluate them alongside classical methods. Our experiments provide new insights into the limitations of current approaches under perceptual aliasing. All datasets, baselines, and evaluation tools are publicly released to foster consistent and reproducible research in topological mapping.