HetroD: A High-Fidelity Drone Dataset and Benchmark for Autonomous Driving in Heterogeneous Traffic
Yu-Hsiang Chen, Wei-Jer Chang, Christian Kotulla, Thomas Keutgens, Steffen Runde, Tobias Moers, Christoph Klas, Wei Zhan, Masayoshi Tomizuka, Yi-Ting Chen
AI summary
Problem
Existing autonomous driving datasets primarily capture structured, lane-disciplined vehicle-to-vehicle interactions, severely underrepresenting vulnerable road users and complex heterogeneous maneuvers. This gap limits the development and evaluation of robust navigation systems for real-world mixed traffic.
Approach
The authors collected a large-scale drone-captured dataset across six diverse urban locations in Taiwan, providing centimeter-accurate annotations, HD maps, and traffic signal states for over 65,000 agent trajectories. They also developed a unified toolkit to convert the data into standard formats and established benchmarks for motion prediction, planning, and simulation.
Key results
- Curated a 17.5-hour drone dataset with 65.4k trajectories, 70% from vulnerable road users, capturing complex unstructured maneuvers.
- Developed a modular toolkit enabling seamless integration with existing autonomous driving frameworks like ScenarioNet and GPUDrive.
- Demonstrated that state-of-the-art prediction models fail to accurately forecast lateral VRU movements and dense multi-agent interactions.
- Showed that drone-view training data generalizes better across domains than on-board data, while map shifts significantly degrade model performance.
Why it matters
Provides a critical, high-fidelity testbed for developing and evaluating robust autonomous driving systems in complex, VRU-rich heterogeneous traffic environments.
Abstract
We present HetroD, a dataset and benchmark for developing autonomous driving systems in heterogeneous environments. HetroD targets the critical challenge of navi- gating real-world heterogeneous traffic dominated by vulner- able road users (VRUs), including pedestrians, cyclists, and motorcyclists that interact with vehicles. These mixed agent types exhibit complex behaviors such as hook turns, lane splitting, and informal right-of-way negotiation. Such behaviors pose significant challenges for autonomous vehicles but remain underrepresented in existing datasets focused on structured, lane-disciplined traffic. To bridge the gap, we collect a large- scale drone-based dataset to provide a holistic observation of traffic scenes with centimeter-accurate annotations, HD maps, and traffic signal states. We further develop a modular toolkit for extracting per-agent scenarios to support downstream task development. In total, the dataset comprises over 65.4k high- fidelity agent trajectories, 70% of which are from VRUs. HetroD supports modeling of VRU behaviors in dense, het- erogeneous traffic and provides standardized benchmarks for forecasting, planning, and simulation tasks. Evaluation results reveal that state-of-the-art prediction and planning models struggle with the challenges presented by our dataset: they fail to predict lateral VRU movements, cannot handle unstructured maneuvers, and exhibit limited performance in dense and multi-agent scenarios, highlighting the need for more robust approaches to heterogeneous traffic. See our project page for more examples: https://hetroddata.github.io/HetroD/