Sharding makes LLM judges more reliable and secure
Split LLM oversight into shards to cut adversarial errors by 3x
A new arXiv study finds that giving an LLM judge more compute doesn't necessarily make it check more requirements—when one call must return many verdicts, agreement with experts declines across research replications, legal work, and clinical-trial assessments. Sharding, which splits requirements into smaller groups judged by separate calls and then aggregates the verdicts, improves expert agreement while holding the model, evidence, total budget, and per-decision budget fixed. A sharded weaker judge can outperform a more capable holistic judge, even when that holistic judge receives the panel's full budget. Sharding also blunts adversarial exploitation: a best-of-N adversary can vary only the presentation of fixed work and multiply an overloaded judge's acceptance of unmet criteria, but sharding removes that advantage wherever it reduces baseline error. The article notes sharding doesn't address attacks that persuade the judge separately on each criterion; in that setting, debate-style opposition on top of sharding withstands adaptive re-optimization.
- Sharding splits multi-criteria LLM oversight into separate calls, improving expert agreement by 25–30% without extra compute
- Overloaded LLM judges are vulnerable to adversarial manipulation that inflates acceptance of unmet criteria by 8x
- Debate-style opposition on top of sharding blocks adversarial re-optimization of individual criteria
Why It Matters
Fixes a core failure mode in AI auditing and safety pipelines, enabling trustworthy autonomous oversight