Event-LAB: Towards Standardized Evaluation of Neuromorphic Localization Methods
Adam D. Hines, Alejandro Fontan, Michael J Milford, Tobias Fischer
AI summary
Problem
The rapid expansion of neuromorphic localization research has fragmented codebases, dependencies, and data formats, making reliable and fair cross-method comparisons difficult and cumbersome.
Approach
Event-LAB uses a single command-line invocation to automatically download datasets, standardize them to HDF5, generate event frames via configurable parameters, and run multiple VPR and SLAM baselines with reproducible dependency management.
Key results
- Unified single-command framework for multi-dataset event-based localization benchmarking
- Demonstration that event count and time window parameters cause large performance variations
- Reconstruction-based frame generation consistently outperforms simple event counts
- Automated pseudo ground-truth generation and batch execution for reproducible analysis
Why it matters
It equips the neuromorphic robotics community with a streamlined, reproducible benchmarking platform to fairly evaluate and compare emerging localization algorithms.
Abstract
Event-based localization research and datasets are a rapidly growing area of interest, with a tenfold increase in the cumulative total number of published papers on this topic over the past 10 years. Whilst the rapid expansion in the field is exciting, it brings with it an associated challenge: a growth in the variety of required code and package dependencies as well as data formats, making comparisons difficult and cumbersome for researchers to implement reliably. To address this challenge, we present Event-LAB: a new and unified frame- work for running several event-based localization methodologies across multiple datasets. Event-LAB is implemented using the Pixi package and dependency manager, that enables a single command-line installation and invocation for combinations of localization methods and datasets. To demonstrate the capa- bilities of the framework, we implement two common event- based localization pipelines: Visual Place Recognition (VPR) and Simultaneous Localization and Mapping (SLAM). We demonstrate the ability of the framework to systematically visu- alize and analyze the results of multiple methods and datasets, revealing key insights such as the association of parameters that control event collection counts and window sizes for frame generation to large variations in performance. The results and analysis demonstrate the importance of fairly comparing methodologies with consistent event image generation param- eters. Our Event-LAB framework provides this ability for the research community, by contributing a streamlined workflow for easily setting up multiple conditions.