ML evaluation harnesses plagued by 3 root causes: study finds 61.7% of issues

A new paper from researchers at Queen's University and other institutions takes a hard look at the software that runs ML model evaluations—so-called evaluation harnesses. By studying 57 open-source and industrial harnesses and classifying 16,560 operational issues, they uncover a clear pattern: most problems happen when integrating external components. The Specification stage, where harnesses wire together models, datasets, and scoring judges, accounts for 41.4% of all issues. Root causes vary by workflow stage—environment incompatibility and external dependency breakage dominate provisioning (36.2% of its issues), while algorithmic error and validation gaps are the main culprits in assessment (25.9% and 22.5% respectively).

The three most frequent root causes overall are unimplemented features (24.3%), documentation gaps (20.3%), and missing input validation (17.2%), together responsible for 61.7% of classified issues. These span both defects in existing functionality and missing capabilities that block intended workflows. The authors also propose a five-stage harness model—Specification, Provisioning, Execution, Assessment, and Reporting—as a framework for diagnosing and improving harnesses. They argue that the field needs “evaluation engineering” as a distinct software engineering concern, analogous to how DevOps emerged from operations challenges. The paper is available on arXiv (2605.24213).