AgentLens evaluates coding agents on full trajectory, not just pass/fail

AgentLens, introduced by a team of researchers including Andrey Podivilov and Vadim Lomshakov, is an open-source benchmark designed to evaluate coding agents on the full trajectory of their interactions. Traditional code-agent benchmarks reduce performance to a single binary outcome — did the task pass or fail? AgentLens argues that real users experience the entire process: how the agent follows instructions, uses tools, verifies its own work, recovers from mistakes, and communicates along the way. By evaluating the whole trajectory, it provides a much richer picture of agent capability and behavior.

Technically, AgentLens pairs formal verification (for tasks with objective checks) with LLM-written trajectory reviews and side-by-side comparisons. Each run yields a readable explanation of why a particular score was assigned, making the benchmark useful beyond model rankings. The creators use it to diagnose model behavior, compare successive versions of their own agent, and catch product regressions in a nightly evaluation pipeline. The benchmark is released as open source, enabling the broader AI community to adopt more nuanced evaluation standards for coding agents.