FlightBench: Benchmarking Learning-Based Methods for Ego-Vision-Based Quadrotors Navigation
Shu'ang Yu, Chao Yu, Feng Gao, Yi Wu, Yu Wang
AI summary
Problem
Head-to-head comparisons between learning-based and optimization-based ego-vision navigation methods are scarce, obscuring their true strengths and limitations. Furthermore, the lack of quantifiable scenario difficulty metrics hinders fair and reproducible evaluation.
Approach
We introduce FlightBench, an open-source benchmark that evaluates representative learning-based and optimization-based methods across diverse 3D scenarios. We develop three quantitative difficulty metrics to categorize test cases and validate performance through simulation and real-world hardware-in-the-loop experiments.
Key results
- First unified open-source benchmark for comparing learning-based and optimization-based ego-vision navigation
- Three quantitative task difficulty metrics (traversability obstruction, view occlusion, angle-over-length) to standardize scenario evaluation
- Learning-based methods excel in high-speed flight and inference speed but struggle in sharp-turn and occluded scenarios
- Real-world hardware-in-the-loop experiments confirm simulation trends and highlight latency randomization's impact on policy robustness
Why it matters
Provides researchers and developers with a standardized platform to objectively evaluate and advance vision-based quadrotor navigation algorithms under realistic, quantified difficulty levels.
Abstract
Ego-vision-based navigation in cluttered environ- ments is crucial for mobile systems, particularly agile quadrotors. While learning-based methods have shown promise recently, head-to-head comparisons with cutting-edge optimization-based approaches are scarce, leaving open the question of where and to what extent they truly excel. In this paper, we introduce FlightBench, the first comprehensive benchmark that implements various learning-based methods for ego-vision-based navigation and evaluates them against mainstream optimization-based baselines using a broad set of performance metrics. More importantly, we develop a suite of criteria to assess scenario difficulty and design test cases that span different levels of difficulty based on these criteria. Our results show that while learning-based methods excel in high-speed flight and faster inference, they struggle with challenging scenarios like sharp corners or view occlusion. Analytical experiments validate the correlation between our difficulty criteria and flight performance. Moreover, we verify the trend in flight performance within real- world environments through full-pipeline and hardware-in-the- loop experiments. We hope this benchmark and these criteria will drive future advancements in learning-based navigation for ego-vision quadrotors. Code and documentation are available at https://github.com/thu-uav/FlightBench.