Unveiling Non-Reproducibility in LiDAR-Inertial Odometry
Hongqian Huang, Meng Zhang, Jianchen Hu, Xiaohong Guan
AI summary
Problem
Despite standardized environments and shared code, modern LiDAR-inertial odometry systems still produce inconsistent localization results under identical conditions, undermining fair algorithm evaluation.
Approach
The authors empirically analyze popular LIO frameworks to identify five specific non-deterministic code patterns, provide deterministic fixes for each, and introduce a systematic binary-search procedure to locate such issues in any LIO pipeline.
Key results
- Identification of five distinct non-deterministic implementation sources in modern LIO systems
- Demonstration that resolving these patterns eliminates accuracy variations exceeding one meter across repeated runs
- Finding that non-reproducibility peaks with low-vertical-resolution LiDARs or in geometrically degenerate scenes
- Development of a general binary-search procedure to systematically locate non-deterministic code in LIO pipelines
Why it matters
Enables fair benchmarking and improves the reliability of LiDAR-inertial odometry systems for robotics and autonomous navigation research.
Abstract
This letter presents empirical research on the non- reproducibility of light detection and ranging sensor (LiDAR)- inertial odometry (LIO) systems. Although the LIO community has made commendable efforts toward reproducible localization accuracy, noteworthy non-reproducibility remains, thus hindering a fair evaluation of method effectiveness. To better understand such non-reproducibility, we first define non-reproducibility and introduce a quantitative criterion to identify noteworthy non- reproducibility. We then propose five significant non-deterministic implementations that are included in state-of-the-art LIO sys- tems and present solutions for modifying these non-deterministic implementations into deterministic ones. A general procedure is also introduced to identify and pinpoint non-deterministic imple- mentations, regardless of whether they are covered in this let- ter. Extensive experiments demonstrate that the non-deterministic implementations are the major or potentially sole causes of non- reproducibility under constant experimental conditions. Addition- ally, the non-reproducibility is noteworthy in datasets obtained from low-vertical-resolution LiDARs or recorded in geometrically degenerate scenes.