Beyond Scalar Rewards: Distributional Reinforcement Learning with Preordered Objectives for Safe and Reliable Autonomous Driving

A research team from institutions including the Karlsruhe Institute of Technology has published a paper proposing a fundamental shift in how reinforcement learning (RL) agents for autonomous driving are trained. The core problem they address is the standard practice of combining multiple objectives—like safety, efficiency, and comfort—into a single scalar reward via weighted summation. This often leads to policies that sacrifice critical safety constraints to optimize for a higher overall score. Their solution is the Preordered Multi-Objective Markov Decision Process (Pr-MOMDP), which explicitly defines a hierarchy, or preorder, among reward components, ensuring safety is prioritized over other goals.

To make this preorder actionable, the researchers developed a novel algorithm based on distributional RL. Instead of comparing the expected value of actions, their method uses a new pairwise comparison metric called Quantile Dominance (QD) to evaluate the full distribution of potential returns for each objective. This allows the AI to identify a subset of optimal actions that are not dominated by others across all prioritized goals. The framework was implemented using Implicit Quantile Networks (IQN) and tested in the high-fidelity Carla driving simulator.

The results demonstrated a tangible improvement over existing methods. Compared to standard IQN and ensemble-IQN baselines, policies trained with the Pr-MOMDP framework achieved higher task success rates while statistically significantly reducing critical failures like collisions and off-road events. By ensuring the AI's decision-making respects a predefined priority of objectives, this work represents a concrete step toward more robust and trustworthy autonomous driving systems, where safety is not negotiable for marginal gains in speed or comfort.