ScarfBench Just Exposed the Ugly Truth About AI Agents and Java Migrations

Every vendor demo shows an AI agent one-shotting a framework migration in ninety seconds. IBM Research just published a benchmark that suggests the reality is closer to a 90% failure rate — and the agents themselves don’t know when they’ve failed. That’s the headline finding from ScarfBench, the first serious attempt to measure whether coding agents can actually modernize enterprise Java applications, and it should make anyone selling “AI-powered legacy modernization” sweat.

Why ScarfBench Is Different From Every Other Coding Benchmark

IBM Research introduced ScarfBench (Self-Contained Application Refactoring Benchmark) as an open evaluation for AI agents on cross-framework migration tasks across Spring, Jakarta EE, and Quarkus. The benchmark contains 34 applications, 102 framework implementations, 204 migration tasks, roughly 151K lines of code, about 2,000 source and test files, and 1,331 expert-written tests. Unlike SWE-Bench-style benchmarks that grade generated code against a reference, ScarfBench requires the migrated application to build, deploy, and pass behavioral validation.

That distinction matters because framework migration is where most enterprise AI budgets go to die. Translating an @Autowired annotation is trivial; preserving the runtime semantics of dependency injection, persistence configuration, JPQL queries, and framework descriptors across an entire application is not. If you’re a bank or insurer running a Spring-to-Jakarta initiative, you don’t care whether the agent produced code that compiles — you care whether the app still passes your regression suite in production. ScarfBench is the first widely available benchmark that grades on that outcome, and it’s going to become the number the modernization vendors get asked about on every RFP.

The 10% Ceiling Nobody Wants to Talk About

According to IBM’s leaderboard, even the strongest current frontier agents achieve less than 10% behavioral success on ScarfBench. Compile success consistently exceeds deploy success, which in turn exceeds behavioral success — meaning “the code built” is a wildly misleading proxy for “the migration worked.” Migration difficulty also varies sharply by target framework, with Jakarta EE proving particularly brutal.

Anyone shipping an AI modernization pipeline needs a validation layer that is entirely independent of the agent doing the work. If you’re a platform team piloting agentic migration on a monolith, treating a green build as “done” is how you end up with a staging environment full of applications that start cleanly and then throw NoSuchBeanDefinitionException under the first real load test. This is exactly the tradeoff at the heart of AI agents versus deterministic AI automation: agents are flexible enough to attempt the migration at all, but rigid, deterministic verification is what keeps them honest. Expect the next wave of enterprise agent products to look less like “chat with your codebase” and more like agent-plus-verifier pipelines with mandatory human sign-off gates.

Agents Are Confidently Wrong About Their Own Work

The most damning finding in the report is about self-assessment. IBM’s team compared agent-reported outcomes against independent build verification and found that Claude Code reported successful builds for 29 out of 30 whole applications — but only 22 of those actually built. The one application the agent classified as failed built correctly on independent verification.

That’s a dangerous failure mode. If you’re wiring an agent into a CI/CD pipeline and letting it self-report status, you’re building a system whose confidence signal is uncorrelated with truth. Imagine a mid-sized fintech running an overnight batch job to migrate 200 microservices from Spring Boot to Quarkus for cloud cost reasons — a system trusting agent self-assessment could greenlight a fleet of broken services before a human ever looks at them. For teams operating in regulated industries like fintech and banking, where audit trails and behavioral guarantees are non-negotiable, this alone justifies mandatory independent test runs on every agent-generated PR. Within twelve months, “verifier as a service” — external, deterministic build-and-test harnesses that grade agent output — becomes a distinct product category, not a feature.

The Work Isn’t Code Translation, It’s Dependency Choreography

ScarfBench’s process traces reveal something that anyone who has done real migration work already knows but that agent marketing consistently ignores. The most frequently visited layers were Configuration, Web, Database, and Service, with common transitions between Configuration ↔ Web and Service ↔ Database. In other words, agents don’t march linearly through source files — they bounce back to configuration artifacts repeatedly as they resolve framework differences.

And many of the failures had nothing to do with code at all. IBM’s team documented Docker cache inconsistencies, port connectivity problems, and Maven wrapper issues as recurring blockers. That’s a hard lesson for anyone designing modernization tooling: the agent needs to reason about the build environment, not just the source tree. If you’re evaluating an AI-integrated software solution that promises “automated legacy modernization,” ask the vendor how their agent handles a stale Docker layer or a broken mvnw — because that’s where the real hours go. Expect the next generation of migration agents to ship with sandboxed environment managers as a first-class component rather than an afterthought.

FAQ

Q: What is ScarfBench? A: ScarfBench (Self-Contained Application Refactoring Benchmark) is an open benchmark from IBM Research for evaluating AI coding agents on enterprise Java framework migration across Spring, Jakarta EE, and Quarkus. It grades agents on whether migrated applications actually build, deploy, and pass behavioral tests — not just whether the generated code looks reasonable.

Q: Why do frontier agents perform so poorly on it? A: Per IBM’s leaderboard, current agents achieve under 10% behavioral success because framework migration requires reasoning about dependency injection, persistence, configuration, and runtime environments — not just source-level translation. Agents also spend disproportionate effort revisiting configuration layers and struggle with environmental issues like Docker caches and Maven tooling.

Q: Can I trust an AI agent to tell me when a migration is done? A: Based on ScarfBench’s findings, no. Claude Code self-reported 29 of 30 whole-application migrations as successful, but independent verification found only 22 actually built. Any production pipeline needs deterministic build-and-test verification separate from the agent doing the work.

Key Takeaways

• Treat any AI modernization pipeline without an independent build-and-test verifier as unfit for production — agent self-assessment is measurably unreliable on ScarfBench data.
• When evaluating vendors, ask specifically for behavioral success rates on ScarfBench or a comparable benchmark; compile-only metrics will overstate migration quality.
• Budget disproportionate engineering time for configuration and environment plumbing on any Java modernization project, since that’s where ScarfBench shows agents actually spend their effort.
• Jakarta EE migrations are the current hard mode for agentic tooling — teams targeting it should pilot on isolated services before committing to a fleet-wide rollout.
• Expect “verifier-as-a-service” tooling and sandboxed build environments to become standard components of enterprise agent stacks within the next year.