When LLMs get significantly worse: A statistical approach to detect model degradations

Researchers have developed a new statistical framework to definitively detect when large language models (LLMs) degrade in performance, even by tiny margins. The method, based on McNemar's test, can confidently attribute performance drops as small as 0.3% to actual model degradation rather than random evaluation noise. This is crucial for monitoring models after optimizations like quantization, which can introduce subtle errors. The tool is implemented on top of the popular open-source LM Evaluation Harness.