NuRisk: A Visual Question Answering Dataset for Agent-Level Risk Assessment in Autonomous Driving
Yuan Gao, Mattia Piccinini, Roberto Brusnicki, Yuchen Zhang, Johannes Betz
AI summary
Problem
Current VLM-based risk assessment relies on static images and qualitative judgments, lacking the spatio-temporal reasoning needed to evaluate how risks evolve for individual agents over time.
Approach
The authors introduce NuRisk, a VQA dataset with sequential bird’s-eye view images and physics-based quantitative risk metrics for 1.1M agent-level samples, and benchmark/fine-tune VLMs to test and improve their temporal risk reasoning.
Key results
- NuRisk dataset with 2.9K scenarios and 1.1M agent-level quantitative risk samples
- Pre-trained VLMs peak at only 33% accuracy with high latency on spatio-temporal reasoning
- Fine-tuned 7B VLM improves accuracy to 41% and reduces latency by 75%
- Establishes a new benchmark highlighting the gap in quantitative agent-level risk assessment
Why it matters
Provides a critical benchmark and dataset for advancing quantitative, agent-level risk reasoning, directly impacting the safety and reliability of autonomous driving systems.
Abstract
Understanding risk in autonomous driving re- quires not only perception and prediction, but also high- level reasoning about agent behavior and context. Current Vi- sion Language Model (VLM)-based methods primarily ground agents in static images and provide qualitative judgments, lacking the spatio–temporal reasoning needed to capture how risks evolve over time. To address this gap, we propose NuRisk, a comprehensive Visual Question Answering (VQA) dataset comprising 2.9K scenarios and 1.1M agent-level samples, built on real-world data from nuScenes and Waymo, completed with safety-critical scenarios from the CommonRoad simulator. The dataset provides Bird’s-eye view (BEV) based sequential images with quantitative, agent-level risk annotations, enabling spatio–temporal reasoning. We benchmark well-known VLMs across different prompting techniques and find that they fail to perform explicit spatio-temporal reasoning, resulting in a peak accuracy of 33% at high latency. To address these shortcomings, our fine-tuned 7B VLM agent improves accuracy to 41% and reduces latency by 75%, demonstrating explicit spatio-temporal reasoning capabilities that proprietary models lacked. While this represents a significant step forward, the modest accuracy underscores the profound challenge of the task, establishing NuRisk as a critical benchmark for advancing spatio-temporal reasoning in autonomous driving. More information can be found at https://github.com/TUM-AVS/NuRisk.