Delay, Plateau, or Collapse: Evaluating the Impact of Systematic Verification Error on RLVR
False negative errors are harmless, but false positives can kill model performance.
Researchers from ETH Zurich have published a new study examining how systematic verification errors impact Reinforcement Learning with Verifiable Rewards (RLVR), a technique widely used to improve LLM reasoning. Prior work treated verification errors as random and independent across samples, concluding they merely slow training. However, real-world verifiers—like static code checkers—often produce systematic errors. The team set out to test whether these systematic errors pose a greater risk.
Through controlled experiments on arithmetic tasks, they discovered a stark asymmetry: systematic false negatives (labeling correct answers as wrong) behave much like random noise and only delay progress. But systematic false positives (labeling wrong answers as correct) can cause three distinct failure modes—delayed convergence, sub-optimal plateaus, or outright performance collapse. Crucially, the severity depends on the pattern of errors, not the overall error rate, meaning simple error-rate monitoring is insufficient. The findings call for deeper scrutiny of verifier quality in RLVR pipelines.
- Systematic false positives in RLVR can cause performance collapse, while false negatives only slow training.
- The specific pattern of errors (not overall rate) determines severity, making pre-hoc detection difficult.
- Contradicts prior research that assumed verification errors have limited practical impact.
Why It Matters
For AI teams using RLVR, this means verifier design is critical—simple error rates are misleading.