New modular AI code review cuts decision losses by 7.3%
LLM code reviewers fail when policies bias risk probabilities by 16.9%
A six-author team led by Rasvik Kudum at arXiv:2608.02677 shows that today’s LLM code-review interfaces violate a basic principle: risk probabilities should be evidence-based, while action costs should drive decisions. In a study of 15,792 reviewer responses across 720 real patches, swapping a 1:1 cost policy for a 10:1 false-accept rule shifted reported failure probabilities by 13.6–16.9 percentage points for every reviewer tested. Worse, under the high-cost prompt every system made worse decisions than simply rejecting every patch.
The authors propose a modular pipeline that first elicits patch risk independently of any policy, then merges that score with an independent monitor signal, and finally applies explicit costs. Against calibrated reviewer-only baselines, the pipeline improves probability accuracy and—at equal costs—cuts mean loss per issue by 0.073 while accepting 58–68% of patches. At a 10:1 cost ratio it accepts none and matches a reject-all baseline, demonstrating that separating risk, evidence, and policy is both feasible and beneficial.
- Four leading LLM reviewer interfaces showed 13.6–16.9 percentage-point swings in reported failure probabilities when the cost policy changed from 1:1 to 10:1.
- A modular pipeline that separates risk elicitation, independent monitor scoring, and explicit cost policies reduces mean loss per issue by 0.073 while accepting 58–68% of patches at equal cost.
- At high cost ratios (10:1), the pipeline accepts no patches and matches a trivial reject-all baseline, proving policy can warp probability estimates.
Why It Matters
Separating risk estimation from cost policy cuts code-review losses by 7.3% while keeping production-ready patch acceptance rates.