RAG attribution metrics fail to transfer across datasets, study finds

A new paper from Tianyu Ding, Aditya Nannapaneni, and Juan Pablo De la Cruz Weinstein audited eight automatic attribution scorers—lexical, embedding, BERTScore baselines, plus entailment/grounding-trained models (clean/FEVER NLI, MiniCheck)—across three evaluation constructs (provenance/topicality, generated-answer attribution, and fact-check entailment). The central question: does any scorer transfer, i.e., stay within the 95% confidence interval of the best audited scorer on every dataset of a multi-dataset construct? The answer is a resounding no. For the generated-answer attribution construct—which has the most multi-dataset human-labeled coverage (AttributionBench's four source datasets, n=1,610, plus independent HAGRID, n=2,150)—none did.

The per-dataset metric rankings inverted dramatically: Kendall tau = -0.64 (p=0.031) between AttributedQA and LFQA. An off-the-shelf NLI scorer that was best on short-claim AttributedQA (AUROC 0.90) collapsed to AUROC 0.53 (chance) on long-form LFQA, where BERTScore won (0.91). This instability is not a length or truncation artifact. The decision cost is concrete: a naive "best-on-average" rule for choosing an evaluator fails leave-one-dataset-out (mean held-out regret 0.172 AUROC, worse than fixing one scorer). A prompt-based LLM judge avoids the chance-level collapses automatic scorers suffer (no LFQA collapse) but is ~100x costlier, non-deterministic, and not uniformly best—relocating the validation burden rather than removing it. The paper concludes that metric choice must be validated on the target dataset.