Beyond Scalar Rewards: Distributional Reinforcement Learning with Preordered Objectives for Safe and Reliable Autonomous Driving
Ahmed Abouelazm, Jonas Michel, Daniel Bogdoll, Philip Schörner, Johann Marius Zöllner
AI summary
Problem
Traditional reinforcement learning collapses multiple driving objectives into a single scalar reward, losing critical priority information and often violating safety constraints. Existing multi-objective methods fail to enforce these priorities during both learning and decision-making.
Approach
The authors introduce a Preordered Multi-Objective MDP to formally encode reward hierarchies, paired with a distributional RL algorithm that uses a novel Quantile Dominance metric to compare full action return distributions. This enables preorder-guided action selection and training that respects objective precedence without scalar aggregation.
Key results
- Formalizes reward hierarchies via the Preordered Multi-Objective MDP (Pr-MOMDP)
- Introduces Quantile Dominance for distribution-aware pairwise action comparison
- Develops Pr-IQN, integrating preorder-guided optimal action subsets into training and inference
- Demonstrates improved success rates, fewer collisions, and greater policy robustness in CARLA benchmarks
Why it matters
Provides a principled way to enforce safety-critical objective priorities in RL, advancing reliable autonomous driving systems for researchers and industry practitioners.
Abstract
Autonomous driving involves multiple, often con- flicting objectives such as safety, efficiency, and comfort. In reinforcement learning (RL), these objectives are typically combined through weighted summation, which collapses their relative priorities and often yields policies that violate safety- critical constraints. To overcome this limitation, we introduce the Preordered Multi-Objective MDP (Pr-MOMDP), which augments standard MOMDPs with a preorder over reward components. This structure enables reasoning about actions with respect to a hierarchy of objectives rather than a scalar signal. To make this structure actionable, we extend distribu- tional RL with a novel pairwise comparison metric, Quantile Dominance (QD), that evaluates action return distributions without reducing them into a single statistic. Building on QD, we propose an algorithm for extracting optimal subsets, the subset of actions that remain non-dominated under each objective, which allows precedence information to shape both decision- making and training targets. Our framework is instantiated with Implicit Quantile Networks (IQN), establishing a concrete implementation while preserving compatibility with a broad class of distributional RL methods. Experiments in Carla show improved success rates, fewer collisions and off-road events, and deliver statistically more robust policies than IQN and ensemble-IQN baselines. By ensuring policies respect rewards preorder, our work advances safer, more reliable autonomous driving systems.