Traffic Scenario Orchestration from Language Via Constraint Satisfaction
Frieda Rong, Chris Zhang, Kelvin Wong, Raquel Urtasun
AI summary
Problem
Generating precise, controllable, and reactive test scenarios for autonomous vehicle simulation is laborious and difficult to scale with manual programming, while purely data-driven or LLM-based methods struggle with exact controllability and closed-loop reactivity.
Approach
The authors propose a neurosymbolic pipeline that uses an LLM to translate natural language scenario descriptions into declarative constraints, which an SMT solver then satisfies to generate kinematically feasible, closed-loop actor trajectories.
Key results
- Higher orchestration success rate than learning-based baselines across diverse scenario families
- Enables precise closed-loop reactivity to uncontrolled ego-vehicle behavior via iterative constraint solving
- Models actor motion as piecewise polynomials for smooth, kinematically feasible trajectories
- Successfully orchestrates complex out-of-distribution scenarios requiring precise spatial and temporal interactions
Why it matters
Provides a scalable, precise, and controllable framework for generating edge-case driving scenarios, directly benefiting autonomous vehicle safety validation and simulation testing pipelines.
Abstract
Autonomous vehicles (AVs) require extensive test- ing in simulation, but test case generation for driving scenarios is laborious. The desired scenarios are often out-of-distribution and have precise requirements on interactions with the AV policy under test. Manually programming scenarios allows for precise controllability but is difficult to scale. On the other hand, statistical models can leverage compute and data, but struggle with precise controllability when out-of-distribution. We cast scenario orchestration as a constraint-solving problem and present a language-in, simulation-out scenario orchestrator for closed-loop testing AVs. Our approach leverages foundation model reasoning to translate general, natural language descrip- tions into a set of constraints as a scenario representation. This then allows us to leverage off the shelf solvers to solve for actor behaviors which meet precise testing intentions in closed-loop. Under a benchmark of carefully crafted and diverse scenario descriptions, our approach greatly outperforms our baselines in orchestration success rate. We further show that our closed-loop approach is especially important for scenarios which require ego-reactive specifications.