New LLM data auditing pipeline catches 99% of label errors without retraining

A new paper from Yunting Song, Matthew Watson, Peter Grabowski, and Jun Qin proposes a scalable, inference-only data valuation pipeline that addresses the growing bottleneck of data quality in LLM alignment. Instead of relying on Shapley value calculations that require repeated model retraining, the method maps semantic k-nearest neighbor neighborhoods into a directed graph and evaluates each record's utility by measuring zero-shot and one-shot conditional log-likelihood shifts using a reference LLM's probability distribution. This yields predictive influence scores that are translated into localized advantage metrics, allowing the system to pinpoint gradient-conflicting records—those that actively harm model performance or encode hidden contradictions.

The pipeline's effectiveness was demonstrated on two major alignment datasets. On HelpSteer2, the automated audit reduced the manual search space by 99.1%, correctly flagging falsely-labeled records across multiple failure modes. On Anthropic's HH-RLHF dataset, it identified thousands of safety and factual preference inversions in both training and evaluation splits. Crucially, the evaluation split analysis revealed that highly capable models often predicted the safer or more helpful response but were penalized by objectively flawed human ground-truth labels—exposing severe vulnerabilities in current benchmark integrity. This work provides a mathematically grounded diagnostic tool for sanitizing datasets and ensuring the integrity of LLM alignment data.