STMutants: First benchmark for testing PLC software mutation detection

Researchers from the field of software engineering have released STMutants, a curated mutation testing dataset specifically designed for IEC 61131-3 Structured Text (ST) programs—the language used extensively in Programmable Logic Controllers (PLCs) for real-time, safety-critical industrial automation. Published on arXiv, the dataset addresses a long-standing gap: while mutation testing is a standard method for evaluating test-suite effectiveness, no public benchmark existed for ST programs. STMutants contains 110 first-order mutants derived from 11 programs sourced from the OSCAT basic library and other industrial contexts. After screening for observability and equivalence (with strong inter-rater agreement at kappa = 0.87), 108 mutants were retained. These mutants cover seven operator categories adapted from classical taxonomies for the PLC domain, including value, relational, arithmetic, logical, negation, operation insertion/omission, and initialization faults. The construction followed a four-phase methodology: fault-type profiling, syntactic transformation, compilability verification, and manual equivalence screening.

To demonstrate the dataset's utility, the authors evaluated three large language models (LLMs) in a two-phase setting: generating test suites and then predicting whether each mutant would be killed or survive. Across the 108 retained mutants, the models achieved mutation detection accuracies of 86.1%, 94.4%, and 86.1%, with statistical analysis confirming significant performance differences. This benchmark opens the door for reproducible research on automated test generation, mutation analysis, fault localization, and AI-assisted quality assurance for PLC software—a domain where software faults can lead to equipment damage, production loss, or unsafe system behavior. By providing the first publicly available mutation benchmark for ST programs, STMutants enables both academia and industry to systematically evaluate and improve testing techniques for industrial automation.