New Audit Method Catches Bad AI Updates With Zero Labeling

AI models in production are never finished. Companies retrain them, shrink them to run cheaper, or quietly swap in a vendor's newer version. Every one of those changes carries a risk: the new model might be worse than the one it replaced, and nobody notices until customers complain. The standard way to catch that is testing — running the model on examples where humans have marked the right answer. That is slow, expensive, and often skipped.

This paper's key insight is surprisingly simple. The difference in error rate between two models can only show up on inputs where they disagree. And you can find those disagreements without any labels at all — just run both models on incoming traffic and compare their answers. The author builds this into DISCERN, a two-tier protocol. A first tier certifies safe updates purely from unlabeled traffic when the models disagree rarely enough. A second tier labels only a sample of the disagreements, using statistical math that stays trustworthy no matter when you decide to stop checking.

The results are strong. Across more than 14,000 replayed audit streams and 785 update pairs — including fine-tuned language models up to 1.4 billion parameters — the method hit an error rate of 0.0002 against a 5% target, caught 98.6% of genuinely worse updates, and produced zero false alarms. Over half of the benign updates were cleared with no labeling at all. Each audit also produces a machine-checkable evidence record, useful for regulators or post-market monitoring.

The catch: DISCERN only tells you whether the new model is worse than the old one, not whether either is actually good. It also needs enough traffic where the two models disagree, which small or narrow applications may not have. And it is a preprint — promising in replay, but not yet proven inside a real company's pipeline.