AI Agents for Legacy Migration Can't Grade Their Own Work

Scope an AI agent to write your Spring-to-Jakarta EE diffs and you are buying a Tier Two agent: read-only tool access and a transcript view, ₹4L – ₹6L in our AI agent cost guide. Scope one you would let near a merge and you are at Tier Three, ₹14L – ₹18L over three to four months, because write access to a system of record only ships with full tracing and an evaluation set behind it. The diff was always the cheap half. AI agents for legacy migration get sold on the cheap half, and the expensive half is establishing that the migrated application still behaves the way the old one did.

IBM Research published a benchmark for this, ScarfBench, which grades coding agents on cross-framework Java migrations by deploying the migrated application and running behavioral validation against it. Claude Code reported successful builds for 29 of 30 whole applications; independent verification found that 22 had actually built.

Seven applications a pipeline would have carried forward on a status field that was wrong. On the thirtieth, the agent reported failure for an application that had built correctly, so the error runs in both directions. Across the benchmark, behavioral success for current frontier agents sits under 10%.

Why AI agents for legacy migration report success they didn’t earn

The agent’s evidence about its own work is the output of the tools it chose to run, read the way it chose to read them. It invoked a build, scanned the log, found nothing it recognized as fatal, and set the task to done. No step in that sequence is a malfunction.

The code compiles. The application deploys and starts. Then it has to behave the way it did last week, and each of those is strictly harder than the one before it. On ScarfBench, compile success runs ahead of deploy success, and deploy success ahead of behavioral success. The distance between the first gate and the last is the part of the job an agent cannot see from inside its own loop.

A passing build proves the annotations resolved and the classpath is coherent. Whether the transaction boundary you had under Spring’s proxy-based AOP survived the move is a separate question, as is whether a JPQL query that used to hand back rows in insertion order still does. Those surface after the build has already passed — in a load test, or in a support ticket.

Treat an agent’s status field as a log line. The status your pipeline acts on has to come from something the agent cannot write to.

What a Java framework migration actually changes

Rewriting @Autowired is find-and-replace with extra steps. The work sits in the dependency injection graph and the bean lifecycles hanging off it, in persistence configuration and the transaction semantics wrapped around it, in framework descriptors, in startup ordering, in the property overrides that differ per environment.

ScarfBench’s process traces show agents spending their time across the Configuration, Web, Database and Service layers, with the frequent transitions running Configuration to Web and Service to Database. Agents return to configuration each time a framework difference surfaces, rather than walking the source tree once.

Migration effort tracks how much of your runtime behavior is decided outside your Java files. Two applications of the same line count can carry very different amounts of that configuration surface, and the line count will not tell you which is which.

So the thing to enumerate before anyone prices the work is that surface. Every deployment descriptor. Every @Configuration class that builds beans conditionally on a profile. Every environment-specific properties file that overrides behavior at startup. Every JNDI lookup and container-managed datasource, with its pool settings. Every scheduled task and the trigger it hangs off. Every filter and interceptor chain, and the order they run in. Then the queries, JPQL and native, that quietly assume one provider’s defaults.

That inventory is the scope. Price against it.

What breaks first: Docker caches and Maven wrappers

ScarfBench’s authors documented stale Docker layer caches and broken Maven wrapper invocations as recurring blockers. Neither is a code problem.

The damaging part is what the agent does next. It reads an environment failure as a code failure and starts editing source that was already correct, because the file system is the only lever it has. It cannot tell a poisoned cache from a wrong import.

An agent doing migration work needs a disposable, deterministic environment per task: a pinned image, with dependencies resolved from a local mirror, so a slow public repository can never present itself as a broken dependency declaration. Building that is platform engineering, and it has to land before the agent gets a task queue — the same DevOps and cloud setup you would want behind any CI system whose verdict you intend to trust.

Ask a vendor selling automated modernization what their agent does when mvnw fails. If the answer is about the model, the answer is no.

How do you prove a migrated application still works?

Build and deploy from a clean checkout, then run the behavioral suite against both the migrated application and the pre-migration one. A person reads the comparison and signs it. The thing doing that work runs outside the agent’s process. It holds its own credentials, so the agent cannot reach the report. Its verdict comes from recorded behavior of the old system. And it is the only thing permitted to mark a migration task complete.

Next to it, a full trace of every run: every tool call with its arguments, every command with its exit code. When a service starts behaving oddly after a merge that passed, the trace is how anyone works out which agent-authored change to look at first. That is the same tracing and human-handoff machinery any production agent needs, and on an AI agent build it gets scoped before the agent is allowed to act.

Do you have the regression tests this needs?

A verifier needs an oracle: a definition of correct behavior that does not come out of the code being changed. ScarfBench ships 1,331 expert-written tests for exactly that reason. The grading standard arrived with the benchmark, independent of the applications being migrated.

Check what your existing tests actually assert on. Tests that pin internal method behavior cannot serve as the oracle here — they will change when the framework changes, and a test you have to edit mid-migration has stopped being evidence of anything.

That makes characterization the first piece of work, before a single migration task is written. Capture real requests and responses at the API boundary, record the database state either side of each one, and turn that into a suite you can run against the old build and the new one. That suite is what the verifier grades against.

Point the agent at the characterization suite before you point it at the migration. A wrong characterization test fails loudly against a system whose behavior you already know. A wrong migration fails quietly against one you don’t.

If there is no appetite for building that suite, do the migration by hand, one module at a time, with a named engineer reading every change.

How to scope an agent-assisted migration

Take one service whose rollback path you have already run at least once. Stand the verifier up before the agent gets near it, then have a named engineer read the diff and the trace before the merge. The gate is a human signature on a verified result, and it stays that way until the behavioral pass rate on your own services argues for loosening it.

Much of a framework migration is deterministic and should stay that way. Mechanical transformations like import rewrites and annotation mappings belong in scripted recipes that produce identical output on every run and can be reviewed once. Reserve the agent for the parts where a rule cannot be written: reconciling two configuration models, working out why a bean initializes too late to be injected. That is the agents versus deterministic automation decision applied to a migration. Every task routed to the agent rather than a recipe adds a verification run and a diff review to the critical path.

Scope the whole thing as an agent engagement: tool definitions, guardrails, an evaluation set, a verifier, a trace store. Those line items are what moves the cost of agent development from one tier to the next, and a migration harness carries all of them.

Migrations we will not take on

A pipeline that migrates a fleet of services overnight on the agent’s own report of success. The self-assessment gap is the reason, and no prompt change closes it, because the agent’s confidence and the agent’s correctness come out of the same process.

A single general modernization agent pointed at an entire monolith. Framework migration is per-module work with a verification step between each module, and an agent handed the whole tree will burn its context re-reading configuration it has already been through.

A migration where nobody owns the regression suite. If the answer to “what proves this still works” is a reviewer’s judgment on a pull request, that review is the only verifier in the system, and it is grading a diff too large for anyone to hold in their head.

And the migration itself, when the risk it retires is smaller than the risk it creates. An unsupported framework is a genuine liability. Containing it behind a locked-down network boundary is also a real answer to that liability, and for an application that is stable and that nobody is asking you to change, it is the cheaper one. Which of the two you are buying is a discovery question. Settle it before a migration budget exists.