LLM test coverage and mutation scores: useful only in regression contexts, study finds

Researchers from the University of Toronto (Zhao, Zhou, Cohen) conducted a large-scale replicability study of two seminal works by Inozemtseva et al. and Papadakis et al., this time focusing on LLM-generated test suites. Their paper, accepted at ISSTA 2026, examines whether code coverage and mutation scores — common proxy metrics for test effectiveness — correlate with real-bug detection in tests created by a diverse set of large language models. The prior studies found that for human-written tests, those correlations largely vanished once test suite size was controlled. This new work diverges substantially: the authors find little evidence that test suite size is a dominant confounder for LLM-generated tests, suggesting the dynamics differ.

The key insight is context-dependence. In regression-style settings where the code provided to the LLM can reasonably be assumed bug-free, coverage and mutation scores provide meaningful signals when comparing across models. However, in common scenarios where the code-under-test may already contain bugs and the goal is to expose them, these proxy metrics become unreliable. The study provides actionable guidance: researchers and practitioners evaluating LLM-based test generation should choose their evaluation metrics based on the specific testing context, not blindly rely on coverage or mutation. This is a significant finding for the software engineering community as LLMs are increasingly used for automated test generation.