AI Evaluation Scores Are 'Perishable' – Paper Proposes Expiration Dates

A new paper on arXiv by Sankalp Gilda and Shlok Gilda challenges the way we interpret AI evaluation scores. The authors argue that evaluation methodologies for language models increasingly combine multiple signals—automated metrics, LLM-as-judge ratings, human assessments, and benchmark suites. When these are aggregated via averaging, a phenomenon they call 'trust inflation' emerges: the overall confidence exceeds the reliability of the weakest signal. To counter this, they propose treating evaluation scores as epistemic claims with three properties: formality (human evaluation provides stronger evidence than automated metrics), scope (a benchmark result applies only to the tested distribution, not universally), and validity windows (results expire as contamination accumulates and distributions shift). The paper draws on chain-of-thought analysis, possibilistic logic, and algebraic theory to establish weakest-link aggregation as the conservative endpoint of a parameterized operator family controlled by a single pessimism parameter.

To illustrate the cost of mean aggregation, the authors apply their framework to the public HELM leaderboard across 54 frontier models on ten scenarios. They found that the top-five models ranked by mean score and by weakest-link are completely disjoint—meaning the models celebrated under average-based rankings are entirely different from those that survive a weakest-link test. This stark result underscores how current averaging practices can produce misleading leaderboards. The paper advocates for evaluation results to carry explicit metadata: a formality tier (e.g., human vs. automated), a scope declaration (e.g., specific distribution or domain), and an expiration date (after which the result is no longer valid). By making the epistemic status transparent, practitioners can avoid over-relying on scores that have degraded or are not appropriate for their use case. The paper was published at the Fifth Workshop on Generation, Evaluation and Metrics (GEM), ACL 2026, in San Diego.