The 10% Problem: Why AI Agents Still Can't Modernize Your Enterprise Java Stack

Frontier AI coding agents can pass programming interviews, ship pull requests, and refactor toy projects on demand — but ask one to migrate a real Spring application to Jakarta EE, and fewer than one in ten attempts will actually preserve the application’s behavior. That’s the uncomfortable finding from ScarfBench, a new open benchmark from IBM Research that measures what happens when AI agents are pointed at the exact kind of work enterprise buyers most want to automate: framework migration in Enterprise Java.

For executives who’ve been sold on the promise that AI agents will slash modernization budgets, ScarfBench is a reality check. And for solutions teams building custom AI for enterprise use cases, it’s a roadmap for where the real engineering value lies — because it isn’t in the code translation.

Why Framework Migration Breaks the AI Agent Story

According to the ScarfBench team at IBM Research, even the strongest current agents achieve less than 10% behavioral success on cross-framework Java migrations. The benchmark spans 34 applications, 102 framework implementations, 204 migration tasks, roughly 151K lines of code, and 1,331 expert-written tests across Spring, Jakarta EE, and Quarkus.

That gap matters because modernization is one of the largest and most expensive activities enterprise engineering organizations undertake. If AI can only reliably finish 1 in 10 of these projects, the business case for “just point an agent at the monolith” collapses. Buyers need to understand that a compilable output is not a working application — and the two are being conflated in vendor demos every day.

Imagine you’re the CTO of a mid-sized insurance company sitting on a decade-old Spring codebase, and a vendor promises AI-driven migration to Quarkus for cloud readiness. ScarfBench suggests that without independent build, deploy, and behavioral validation baked into the process, you’re likely paying for output that looks correct but silently breaks in production. Within the next 18 months, “agent-assisted modernization” contracts will start including behavioral-parity SLAs, and vendors who can’t meet them will lose enterprise deals to those who can.

The Overconfidence Problem Nobody Is Pricing In

According to IBM Research’s evaluation, Claude Code reported successful builds for 29 out of 30 whole applications. Only 22 of those actually built. And the single application the agent flagged as failed? It built correctly.

Agent self-assessment, in other words, is not a reliable signal. That changes the economics of AI-assisted modernization: if you can’t trust the agent’s own “done” signal, you need a parallel verification layer — CI pipelines, containerized deploy tests, and behavioral test suites — running against every migration attempt. That’s real infrastructure work, and it’s exactly the kind of scaffolding most off-the-shelf AI tools don’t ship with.

If you’re a platform team piloting a coding agent for legacy modernization, you cannot skip the verification harness. You need expert-written tests, independent build validation, and a rollback path — the same discipline you’d apply to any high-risk deployment. The distinction between custom AI and off-the-shelf SaaS AI goes straight to the bottom line here: the SaaS tool gives you the agent; a custom build gives you the verification loop around it.

Expect the next wave of enterprise AI procurement RFPs to explicitly demand third-party validation of agent outputs, not just model benchmarks. The vendors treating this as an afterthought will get filtered out fast.

Configuration, Not Code, Is the Real Bottleneck

One of the most useful findings in ScarfBench is where agents actually spend their effort. Per IBM Research, the most frequently revisited layers during migration were Configuration, Web, Database, and Service — with configuration dominating. Common back-and-forth transitions included Configuration ↔ Web and Service ↔ Database. Migration, the team concludes, is an iterative dependency-resolution process rather than a simple source-to-source transformation.

AI modernization isn’t a translation problem — it’s a systems problem. The agent has to reason about dependency injection, persistence configuration, framework descriptors, build tooling, and runtime environments — often circling back to the same artifacts multiple times as new failures surface.

For a bank migrating a Spring-based transaction service to Jakarta EE, this means the AI agent’s value isn’t in rewriting @Autowired to @Inject. It’s in navigating the cascading changes across pom.xml, persistence.xml, application properties, and Docker configurations without losing track of state. That kind of orchestration is closer to what a purpose-built AI agent architecture is designed to deliver — long-running, tool-using, environment-aware — than what a chat-based coding assistant offers.

Here’s the prediction: the winning AI modernization products of 2027 will look less like coding copilots and more like infrastructure automation platforms with LLM reasoning bolted on. The competition will shift from code generation to dependency-graph navigation.

Environment and Tooling Failures Are the Hidden Cost

The ScarfBench team also found that agents frequently struggled with issues that have nothing to do with source code: Docker cache inconsistencies, port connectivity problems, and Maven wrapper failures. These operational concerns delayed validation even when the code migration itself was largely complete.

Enterprise buyers rarely see this in a demo. A migration project doesn’t fail because the AI wrote bad Java — it fails because the CI runner can’t reach the test database, or the Docker layer cache went stale, or the Maven wrapper picked the wrong JDK. These are the failure modes that eat weeks of engineering time on real modernization projects.

If you’re running a modernization program, your AI investment needs to be paired with investment in the surrounding enterprise integration and API infrastructure — the reliable glue that lets the agent actually observe whether its changes worked. Without that, you’re paying for intelligent guesses.

FAQ

Q: What is ScarfBench? A: ScarfBench (Self-Contained Application Refactoring Benchmark) is an open benchmark from IBM Research for evaluating AI agents on cross-framework migration tasks in Enterprise Java. It covers migrations across Spring, Jakarta EE, and Quarkus, and it measures whether migrated applications actually build, deploy, and preserve behavior — not just whether the generated code compiles.

Q: Why is enterprise framework migration harder than typical coding benchmarks? A: Framework migration requires translating framework semantics, not just source code. According to IBM Research, success depends on preserving behavior across dependency injection, persistence configuration, queries, and framework descriptors, plus adapting build systems and runtime dependencies. Small mistakes in any of these can prevent successful deployment even when the code looks correct.

Q: Should enterprises still use AI agents for modernization given these results? A: Yes, but with realistic expectations and independent verification. The ScarfBench findings show frontier agents can automate substantial portions of the migration process, but reliable validation and architectural reasoning remain critical. Treat agent output as a draft that requires build, deploy, and behavioral testing before it counts as complete.

Key Takeaways

• Enterprise buyers evaluating AI modernization vendors should demand behavioral-parity metrics, not compile-success rates — the two diverge dramatically in real projects.
• Any serious agent-based migration program needs an independent verification harness; the agent’s own “done” signal is not trustworthy per IBM Research’s findings.
• Budget for configuration and environment work, not code translation — that’s where the effort actually accumulates during framework migrations.
• The next generation of modernization tools will compete on dependency-graph reasoning and infrastructure awareness, not raw code generation quality.
• Teams piloting AI agents on Enterprise Java should start with focused migration tasks and measurable test suites before attempting whole-application migrations, where success rates remain especially low.