VeriHarness boosts LLM agent repair by 44 points with structured feedback
Structured feedback triples repair success in AI agents by specifying admissible alternatives.
A new paper from Jaideep Ray and Ankit Goyal introduces VeriHarness, a code-controlled agent loop designed to standardize how LLM agents receive and act on external validation feedback. Unlike typical retry loops that pass raw diagnostic errors, VeriHarness allows researchers to control feedback structure—specifically whether it includes the failure location, observed value, and admissible alternatives. The team tested this across 50 paired TextWorld games under a strict four-call cap, comparing raw diagnostics to structured feedback with all three fields.
Results show dramatic improvements: Qwen2.5-Coder-14B went from 14/50 to 36/50 terminal success (+44 percentage points), and Llama-3.1-8B from 8/50 to 29/50 (+42 points). Ablations reveal that admissible alternatives drive most of the gain—feedback with only location and observed value barely outperforms raw diagnostics. Interestingly, presenting the same repair information in prose vs. keyed JSON yields nearly identical success, suggesting that the content matters far more than the syntax. These findings hold across different call budgets and sampling settings, offering a clear design principle for building more reliable AI agents.
- VeriHarness framework provides a standardized loop for comparing feedback formats in LLM agent repair.
- Adding admissible alternatives raises terminal success by 44 pp for Qwen2.5-Coder-14B and 42 pp for Llama-3.1-8B.
- JSON formatting offers no advantage over prose; the content of feedback (especially acceptable values) is what matters.
Why It Matters
Practical guide for building self-correcting AI agents: tell them exactly what's allowed, not just what's wrong.