On-Device Self-Supervised Learning of Visual Perception Tasks Aboard Hardware-Limited Nano-Quadrotors

Sub-50 g nano-drones are gaining momentum in both academia and industry. Their most compelling applications rely on onboard deep learning models for perception despite severe hardware constraints (i.e., sub-100 mW processor). When deployed in unknown environments not represented in the train- ing data, these models often underperform due to domain shift. To cope with this fundamental problem, we propose, for the first time, on-device learning aboard nano-drones, where the first part of the in-field mission is dedicated to self-supervised fine- tuning of a pre-trained convolutional neural network (CNN). Leveraging a real-world vision-based regression task, we thor- oughly explore performance-cost trade-offs of the fine-tuning phase along three axes: i) dataset size (more data increases the regression performance but requires more memory and longer computation); ii) methodologies (e.g., fine-tuning all model parameters vs. only a subset); and iii) self-supervision strategy. Our approach demonstrates an improvement in mean absolute error up to 30% compared to the pre-trained baseline, requiring only 22 s fine-tuning on an ultra-low-power GWT GAP9 System-on-Chip. Addressing the domain shift problem via on-device learning aboard nano-drones not only marks a novel result for hardware-limited robots but lays the ground for more general advancements for the entire robotics community. SUPPLEMENTARY MATERIAL Experiment results video: https://youtu.be/blOid4iUFAM