Event Spectroscopy: Event-based Multispectral and Depth Sensing using Structured Light
Christian Geckeler, Niklas Neugebauer, Manasi Muglikar, Davide Scaramuzza, Stefano Mintchev
AI summary
Problem
UAVs operating in dense forests require low-latency depth sensing and accurate spectral data for navigation and ecological monitoring, but current solutions rely on bulky multi-sensor setups that struggle under variable canopy lighting and lack fine depth resolution.
Approach
By projecting modulated structured light wavelengths and exploiting the event camera’s illumination-dependent sensor response, the system reconstructs synchronized depth maps and multispectral reflectivity in a unified, low-latency pipeline.
Key results
- Up to 60% RMSE improvement over commercial depth sensors
- Spectral accuracy matches reference spectrometers and commercial multispectral cameras
- Successful RGB reconstruction and leaf-branch differentiation in a real-world rainforest
- Adding depth data boosts material classification accuracy by over 30%
Why it matters
Enables lightweight, robust, and all-in-one perception for UAVs navigating complex natural environments without relying on multiple heavy sensors.
Abstract
Uncrewed aerial vehicles (UAVs) are increasingly deployed in forest environments for tasks such as environmental monitoring and search and rescue, which require safe navigation through dense foliage and precise data collection. Traditional sensing approaches, including passive multispectral and RGB imaging, suffer from latency, poor depth resolution, and strong dependence on ambient light—especially under forest canopies. In this work, we present a novel event spectroscopy system that simultaneously enables high-resolution, low-latency depth reconstruction with integrated multispectral imaging using a single sensor. Depth is reconstructed using structured light, and by modulating the wavelength of the projected structured light, our system captures spectral information in controlled bands between 650 nm and 850 nm. We demonstrate up to 60% improvement in RMSE over commercial depth sensors and vali- date the spectral accuracy against a reference spectrometer and commercial multispectral cameras, demonstrating comparable performance. A portable version limited to RGB is used to collect real-world depth and spectral data from a Masoala Rainfor- est. We demonstrate color image reconstruction and material differentiation between leaves and branches using this spectral and depth data. Our results show that adding depth (available at no extra effort with our setup) to material differentiation improves the accuracy by over 30% compared to color-only method. Our system, tested in both lab and real-world rainforest environments, shows strong performance in depth estimation, RGB reconstruction, and material differentiation—paving the way for lightweight, integrated, and robust UAV perception and data collection in complex natural environments.