Tiny-DroNeRF: Tiny Neural Radiance Fields Aboard Federated Learning-Enabled Nano-Drones
Ilenia Carboni, Elia Cereda, Lorenzo Lamberti, Daniele Malpetti, Francesco Conti, Daniele Palossi
AI summary
Problem
Nano-drones lack the computational and memory resources to run complex vision tasks like 3D reconstruction, while existing NeRF methods require gigabytes of memory and high-end GPUs, severely limiting their spatial awareness and autonomous capabilities.
Approach
The authors optimize the Instant-NGP NeRF algorithm for extreme memory and compute constraints on a GAP9 ultra-low-power MCU, then use a federated learning scheme to collaboratively train a global model across a swarm of nano-drones without exchanging raw images.
Key results
- 96% memory footprint reduction compared to Instant-NGP
- Only 5.7 dB PSNR loss while maintaining reconstruction quality
- Successful on-device training on a GAP9 MCU under 0.10 W
- Federated swarm training achieves centralized-level accuracy with minimal communication overhead
Why it matters
Enables resource-constrained nano-drones to perform advanced spatial awareness and 3D mapping tasks autonomously, advancing swarm robotics for inspection and search-and-rescue applications.
Abstract
Sub-30 g nano-sized aerial robots can leverage their agility and form factor to autonomously explore cluttered and narrow environments, like in industrial inspection and search and rescue missions. However, the price for their tiny size is a strong limit in their resources, i.e., sub-100 mW microcontroller units (MCUs) delivering ∼100 GOps/s at best, and memory budgets well below 100 MB. Despite these strict constraints, we aim to enable complex vision-based tasks aboard nano-drones, such as dense 3D scene reconstruction: a key robotic task underlying fundamental capabilities like spatial awareness and motion planning. Top-performing 3D reconstruction methods leverage neural radiance fields (NeRF) models, which require GBs of memory and massive compu- tation, usually delivered by high-end GPUs consuming 100s of Watts. Our work introduces Tiny-DroNeRF, a lightweight NeRF model, based on Instant-NGP, and optimized for running on a GAP9 ultra-low-power (ULP) MCU aboard our nano-drones. Then, we further empower our Tiny-DroNeRF by leveraging a collaborative federated learning scheme, which distributes the model training among multiple nano-drones. Our experimental results show a 96% reduction in Tiny-DroNeRF’s memory footprint compared to Instant-NGP, with only a 5.7 dB drop in reconstruction accuracy. Finally, our federated learning scheme allows Tiny-DroNeRF to train with an amount of data otherwise impossible to keep in a single drone’s memory, increasing the overall reconstruction accuracy. Ultimately, our work combines, for the first time, NeRF training on an ULP MCU with federated learning on nano-drones. SUPPLEMENTARY VIDEO MATERIAL Supplementary video can be found at: https://youtu.be/-frFPUBGa0c