Why AI Agent Pilots Stall: Make the Pipeline Predictable

Ask why AI agent pilots stall and you’ll hear that the model wasn’t good enough yet. It almost never is the model. The pilot worked — that’s the reason it got a second meeting. What’s missing is a way to state that version four behaves better than version three, in a form someone other than the person who built it can check. Until that exists, nobody with signing authority puts the agent in front of a customer, and they are right not to.

Why AI agent pilots stall after the demo

A demo is a sample of one. Run the same request through the same agent twice and the model can pick a different tool and hand back a different answer. That is how language models work. No amount of prompt tightening turns it off.

Everything downstream of the demo assumes the opposite. A release process exists because you can run a test twice and get the same result — that is the whole premise of a passing build. An agent breaks it. A test that passed once carries no information about the next run, so the usual evidence is unavailable at exactly the moment somebody has to sign.

What happens instead is a meeting. The team that built the agent says it works well. Whoever owns the risk asks how well, against what, measured how. Nothing on paper answers that. The pilot goes back on the shelf with a note about revisiting it after the next model release.

What replaces a passing test when output changes every run

Give up on making the agent’s output repeatable. Make the release decision repeatable.

The mechanism is a score. You hold a fixed set of tasks, run the candidate version against all of them, and grade each result on whether it reached the right outcome and whether it stayed inside the policy you set. That produces a number. Give the number a floor. The deploy step fails when the score drops below the floor, the same way it fails on a broken unit test, even though nothing inside the run was deterministic.

That turns an argument into a threshold, and the threshold is the harder decision. Set it at perfection and you will never ship, because a real support queue contains cases with no clean right answer, which is why the person working it today does not clear them all either. The defensible floor is what your existing process achieves on the same cases. Clear that, with failures of a recoverable kind, and you have something to take into a risk review.

What an agent evaluation set has to contain

Real cases out of your own systems, each with the outcome a person actually reached. Closed tickets and past reconciliations — whatever the agent is meant to take over.

Synthetic prompts written by the build team encode the same assumptions the agent was built on, so a good score against them predicts nothing. The cases worth including are the ones that went badly: the ambiguous request, the customer who asked for two things at once, the record with a field missing. A model change shows up there first, and it shows up as a slightly different tool choice rather than as an error anyone would catch.

Then the score gets recorded against a version: the model it ran on and the prompt it ran with. A provider updating a model behind your API call is a change you did not make and were not told about, and a recorded score is how you find out it happened. Any AI agent build should hand this set over as an artifact alongside the running agent. Build it during the pilot, out of tickets you already have.

Why every agent run needs a trace you can read back

A trace is the record of one run, and you can keep it: every model call, every tool call with its arguments, every result, and the spend.

With one, a regression is a diff. The run took its usual path until step four, where it called a different endpoint — enough to change one tool description and re-score. Without one, the only way to investigate is to re-run the prompt and hope the failure repeats while you’re watching, with outputs checked by hand because nothing else in the system checks them.

Log the trace from the first pilot run. Retrofitting it means re-running every case you already paid to evaluate.

Who owns each agent once the pilot team moves on

Pilots get built by whoever was curious. Then several teams have something running, each with its own prompt and its own credential, and no register lists any of it. The governance question has no answer available at that point, because nobody can produce the list.

So promotion to production carries a named owner. One row per agent: who owns it, which tools it may call, what it last scored, which model version it’s pinned to. That row is what a risk review asks for. Writing it costs nothing at the moment of creation and becomes a reconstruction job once the agent has been running for a while. Add the row when the agent is created, not when someone asks for the list.

What it costs to take an agent from pilot to production

The pilot is the cheap part. A first agent proven against your real past cases in a sandbox, with no writes to live systems and no guardrail configuration, sits in the ₹1L – ₹1.5L band in our agent scoping guide. Scoped credentials, the approval path, the evaluation set and the tracing belong to the tiers above it, and that is where the budget goes. The distance between those two figures reads as overhead added by cautious engineers. It buys the sign-off the pilot could not get on its own.

Worth asking at that stage whether the job needed an agent at all. A task with fixed rules and a known sequence runs without a model choosing tools at runtime, and a deterministic workflow can be tested the ordinary way, which deletes this whole problem class. Settle agents against automation before anyone writes an eval harness.

The trade-off worth accepting: a narrower agent behind a scored gate over a broader one that has only ever been judged by eye. Less coverage on day one, and slower to widen. It survives a model upgrade without a war room. Widening scope against a passing evaluation set is an ordinary piece of work. Widening it on the strength of a good demo is a bet you will be asked to defend the first time the agent gets something wrong in front of a customer.