HarnessFix Diagnoses LLM Agent Failures, Boosting Reliability by 50%

LLM agents rely on complex harnesses for execution, tool interfaces, and orchestration, but failures often stem from subtle flaws in these harness layers. Existing self-improvement methods make broad, poorly scoped changes because they cannot pinpoint which part of the trajectory or harness caused the failure. To solve this, researchers from the Chinese Academy of Sciences propose HarnessFix, a trace-guided framework that systematically diagnoses agent failures and repairs harnesses. It first converts raw execution traces and harness code into a Harness-aware Trace Intermediate Representation (HTIR), normalizing fragmented evidence and capturing step-level provenance and control flow. HarnessFix then attributes failures to specific trajectory steps and harness layers (e.g., execution environment, tool interfaces, lifecycle orchestration), consolidates recurring issues into actionable flaw records, and maps them to scoped repair operators. Finally, it generates and validates harness patches to reduce target flaws without introducing regressions.

Evaluated on four challenging benchmarks—SWE-Bench Verified, Terminal-Bench 2.0 Verified, GAIA, and AppWorld—HarnessFix improved held-out test performance by 15.2% to 50.0% over initial harnesses, outperforming both human-designed baselines and automatic self-evolution methods. The framework also revealed recurring harness-flaw patterns across the ETCLOVG layers (Execution, Tool, Context, Lifecycle, Observability, Verification, Governance). This work provides a systematic approach to making LLM agents more reliable by addressing the root cause of failures rather than applying ad-hoc fixes. For professionals deploying AI agents in production, HarnessFix offers a path to identify and repair fragile harness components, reducing unexpected failures and improving overall agent robustness.