Fusing Event-Based Camera and Radar for SLAM Using Spiking Neural Networks with Continual STDP Learning

This work proposes a first-of-its-kind SLAM ar- chitecture fusing an event-based camera and a Frequency Mod- ulated Continuous Wave (FMCW) radar for drone navigation. Each sensor is processed by a bio-inspired Spiking Neural Net- work (SNN) with continual Spike-Timing-Dependent Plasticity (STDP) learning, as observed in the brain. In contrast to most learning-based SLAM systems, our method does not require any offline training phase, but rather the SNN continuously learns features from the input data on the fly via STDP. At the same time, the SNN outputs are used as feature descriptors for loop closure detection and map correction. We conduct numerous experiments to benchmark our system against state-of-the-art RGB methods and we demonstrate the robustness of our DVS- Radar SLAM approach under strong lighting variations. MULTIMEDIA MATERIAL Please watch a demo video of our SNN-based DVS-Radar fusion SLAM at https://youtu.be/a7gvZWNHGoI