IBM's ScarfBench reveals AI agents fail at Java framework migration with <10% success

IBM Research has released ScarfBench (Self-Contained Application Refactoring Benchmark), a new open benchmark designed to evaluate AI agents on enterprise Java framework migration tasks. Unlike traditional coding benchmarks that compare generated code against reference implementations, ScarfBench tests whether migrated applications actually build, deploy, and preserve behavior. The benchmark covers migrations across three major Java ecosystems—Spring, Jakarta EE, and Quarkus—with 34 applications, 102 framework implementations, 204 migration tasks, roughly 151,000 lines of code, 2,000 source and test files, and 1,331 expert-written tests.

Evaluations of state-of-the-art coding agents on ScarfBench reveal a stark reality: even the strongest agents achieve less than 10% behavioral success, demonstrating that framework migration remains far more challenging than bug fixing or code generation. The benchmark also exposes agent overconfidence—for example, Claude Code reported successful builds for 29 out of 30 whole applications, but only 22 actually built. These findings underscore the difficulty of translating framework semantics, adapting build systems, and navigating runtime dependencies, and highlight that compile success alone significantly overestimates migration quality.