JobBench benchmark refocuses AI agents on human delegation, not replacement
Despite a decade of agentic AI hype, the best model scores just 45.9% on a benchmark that measures human delegation—not replacement. That gap is the story.
A large team of researchers from leading universities and institutes has introduced JobBench, a benchmark designed to evaluate AI agents on what human experts actually want delegated, rather than what maximizes GDP. Current occupational benchmarks are scoped primarily by economic values, telling a replacement story. JobBench flips this: it covers 130 agentic tasks across 35 occupations, each packaged as a workspace of heterogeneous reference files that require reasoning through cluttered information streams typical of real professional work. Outputs are graded by a fact-anchored chain of rubrics, averaging 35.6 binary criteria per task to ensure precise, objective scoring.
The team evaluated 36 models on JobBench, and the strongest performer—Claude Opus 4.7 under Claude Code—reached only 45.9%. This stark result shows that even leading AI systems struggle with professional delegation tasks. The authors hope JobBench will shift the community's focus from building agents that replace humans to building agents that enhance human capabilities by doing what people actually want delegated. The paper is available on arXiv and is pending DOI registration.
- JobBench introduces expert-defined rubrics with an average of 35.6 binary criteria per task, shifting evaluation from replacement to delegation—a new standard for AI agent testing.
- The top model's 45.9% score reveals a substantial gap between current agentic AI capabilities and real-world human trust requirements, with implications for the $30B+ agent market.
- Hidden risks include potential expert bias in rubric design, oversimplification of nuanced professional tasks, and uncertainty about the model identity ('Claude Opus 4.7'), which may affect benchmark reproducibility.
Why It Matters
JobBench reframes AI evaluation around human delegation, exposing the gap between agentic hype and practical workplace trust.