CoW Scoring: New evaluation framework for LLM agents in real apps
PostgreSQL-level Copy-on-Write isolates agent writes for cheap, granular scoring.
Current evaluation mechanisms for LLM-based agents suffer from low construct validity on application-specific workflows and are often costly to replicate. CoW Scoring solves this by operating directly within the target application's environment, leveraging PostgreSQL's Copy-on-Write technology to snapshot and isolate agent database writes. This allows evaluators to generate granular, operation-level scores without building full replica environments, making iteration cheap and fast. The framework was demonstrated on Plane, an open-source project management platform, where it successfully identified specific failures in the agent's tool surface—leading to targeted fixes that produced measurable improvements across affected models.
The paper, accepted at the ICML 2026 Second Workshop on Agents in the Wild, details a 15-page analysis with 11 figures, including a Python library for implementation. CoW Scoring addresses a critical gap: as agents become more autonomous in production software systems, teams need lightweight, precise ways to localize errors without draining engineering resources. By isolating only write operations (the most risky and often the source of failures), CoW offers a practical path to continuous agent evaluation in real-world CI/CD pipelines.
- CoW Scoring uses PostgreSQL's Copy-on-Write to isolate agent database writes, enabling granular session-/operation-level scores.
- Tested on Plane (open-source project management), it surfaced specific tool surface issues; fixes led to measurable model improvements.
- Accepted at ICML 2026's 'Agents in the Wild' workshop; Python library available on GitHub.
Why It Matters
Cheap, precise agent eval in production environments accelerates safe deployment of LLM agents into real software systems.