AI Agent Permissions: Instructions Are Not a Guardrail

In April an AI coding agent deleted a startup’s production database and three months of volume backups with a single API call. Nine seconds. It hadn’t been hacked and it wasn’t misconfigured — it hit a credential mismatch in a staging environment, decided a delete would resolve it, and held a fully permissioned token that let the call through. Afterward it wrote out, accurately, which of its own safety rules it had broken, including the one instructing it never to guess. That story is written up here.

The rule against guessing sat in the prompt. The token sat in the environment. Only one of those is enforced by a computer.

How an agent causes damage without malfunctioning

The failure looks nothing like a crash. An agent working through a task hits an inconsistency — the record it expected isn’t there, the environment it’s pointed at doesn’t match the credential it was handed — and it has a goal, a tool that clears the inconsistency, and permission to call that tool. So it calls it. Every step is the agent behaving as designed. No exception thrown, no alert. The damage arrives as a successful API response.

Models are trained to be careful with destructive operations, and most of the time they are. Most of the time is the problem. That caution moves with how a request was phrased, and with the model version your provider rolled out on Tuesday without telling you. When it moves, it shows up as slightly different tool choices rather than as errors, which is why a model change needs a regression suite of its own: real tasks with known-good outcomes, scored before launch, re-run after every version bump, with a diff on the tool calls the agent chose.

The one guardrail layer you own

Guardrails get discussed as a property of the model. Of the layers that could hold this line, one belongs to you.

The model layer doesn’t. Providers do real work there and it helps at the margin, but you don’t own the weights and you can’t audit the decision. The framework layer barely does — most give you an on/off switch per tool category, which is a long way from this agent may call this endpoint, with these arguments, up to this limit. The credential is yours, and it is the layer that fails closed.

So the agent gets its own identity rather than borrowing an engineer’s. A separate token per agent, per environment, denied by default, with an allowlist naming the specific routes it may touch. If the job is issuing refunds below a threshold, the token issues refunds below that threshold, and it has no route at all to the environment where the backups live.

Read access is a permission too. An agent that can search across your document stores becomes an extraction path the moment someone can get text into its context, and a supplier’s PDF invoice is text your agent will read. So is a support ticket written by whoever wants a look at the customer table. Scope reads per source and per tenant, and log which documents each run actually retrieved. The question a customer or a regulator eventually asks is never whether the agent could have seen their record. It is whether it did.

Most of that work lands in the API layer rather than inside the agent, which is why your own endpoints need scoped auth and per-token limits — the API and integration layer — before the agent is issued anything. Then run the blast radius question in writing. Assume something in the agent’s context has convinced it to do the worst thing its token permits, and state that thing in one sentence. If the sentence won’t finish, narrow the token until it does.

Which actions stop for a signature

Not all of them. An approval prompt on every action produces an agent nobody uses and a reviewer who clicks approve without reading, because nothing in the queue has ever been wrong before.

Gate on properties of the action rather than on categories of tool:

• It is irreversible, or undone only by someone’s manual effort — deletes, refunds, messages that go out to customers.
• It crosses a spend or value threshold the business owns.
• It touches more than a handful of records in one call.
• It writes to a system of record you couldn’t reconstruct from anywhere else.

What tends to get shipped instead is a prompt telling the agent to ask before it does anything risky. The agent can decline to ask, and nothing stops the call when it does. A real gate sits in the execution path: the agent emits the action as structured data, it lands in a queue, a named person approves it, and the system executes it. For gated operations the agent never holds the credential.

That line also sets the price of the build. An internal agent that writes into your own systems sits in the ₹4L – ₹6L band on our agent scoping guide, and whether it writes at all or only reads moves the figure more than any other decision in the scope.

When the lockdown eats the point of the agent

Scoped hard enough, the agent stops earning its cost. Gate every action, demand a signature at each step, and you have built a slower copy of the manual process with a language model sitting in the middle of it. The token bill is new. The throughput isn’t.

When a scope collapses like that, the work probably never needed an agent. It needed a defined workflow with fixed rules, which is cheaper and more predictable to build, and the agents versus automation trade-off is worth settling before anyone writes a tool definition.

Given the choice, narrow the job before widening the permissions. One task the agent owns end to end, with a token cut to fit it, beats a general-purpose agent with broad access and a nervous engineer watching the log. Settling that is most of what the first week of an AI agent engagement goes on.

Before anyone issues the first token, write the one sentence naming the worst thing that credential permits, and put the name of the person who signs it off underneath.