Stephanie Jarmak's 314-page monograph: coding agents fail at system level, not model level

Stephanie Jarmak's new monograph, "Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Model" (arXiv:2608.13867), makes a critical argument: AI coding agents are typically evaluated as isolated models, but in production they operate as complex systems. Reliability depends on the harness, execution state, retrieval, memory, permissions, review interfaces, and resource allocation—not just the underlying LLM. The 314-page review synthesizes 164 scholarly works, 100 practitioner records, 29 benchmark records, and 17 author-system case records through a structured multivocal review and distributed-systems evidence synthesis.

The central finding: many apparent model failures actually originate elsewhere in the system, and improvements at one layer often fail to improve end-to-end outcomes. To address this, Jarmak contributes a versioned catalog of 206 reliability records—193 gated practices (56 developed in depth) plus 13 research leads—along with an evidence ledger and a framework for dependency and repair asymmetry across the agent lifecycle. The monograph includes runnable evaluation and reliability protocols plus five reusable agent skills with evidence maps. This system-level methodology helps teams design defensible evaluations, distinguish model capability from infrastructure effects, and build systems that recover safely when components fail. For engineers operating coding agents at scale, this is a practical field guide, not just theory.