New SAMT method for autonomous driving testing detects real-world hazards
Injecting faults between modules, not just components, reveals hidden dangers.
A new vision paper by Donghwan Shin (arXiv, June 2026) introduces Safety-Aware Mutation Testing (SAMT), a paradigm shift for evaluating the test adequacy of simulation-based autonomous driving systems (ADS). Traditional approaches either focus on individual component reliability or treat the ADS as a black box, missing the interaction failures that cause the majority of accidents. SAMT addresses this by systematically injecting temporally bounded faults into the messages exchanged between ADS modules, simulating realistic failures like sensor miscommunication or delayed perception signals.
To ensure these mutants represent genuine hazards, Shin proposes deriving mutant generation rules directly from System-Theoretic Process Analysis (STPA), a top-down safety engineering framework. This embeds systems thinking into mutation testing, creating a rigorous mechanism to determine when to stop generating additional test scenarios. SAMT promises to enable automated scenario generation, guide ADS repair, and provide a falsifiable measure of test adequacy—something the community currently lacks. The paper also outlines critical open challenges, including scalability and integrating with existing ADS pipelines.
- SAMT injects faults into inter-module messages (not individual components) to simulate realistic interaction failures in autonomous driving systems.
- Mutant generation rules are derived from System-Theoretic Process Analysis (STPA), a safety engineering framework, ensuring faults represent genuine hazards.
- The approach addresses a critical gap: most ADS accidents stem from component interactions, which existing coverage metrics fail to capture.
Why It Matters
SAMT could make autonomous driving simulations safer by providing a rigorous, hazard-aware way to know when testing is sufficient.