Data Infrastructure for AI Workloads Is the Next Bottleneck

The standard fix for an AI feature that costs too much to run is to go shopping for a cheaper model. Swap the frontier model for a small one. Quantize it, cache the prompt prefix, trim the system message, move the embeddings to a cheaper provider. All of that is real engineering, and all of it is aimed at the one line on the bill that gets cheaper every year whether or not anybody works on it. The data infrastructure for AI workloads underneath the feature is the line nobody opens, and it is the same stack that was built when the only thing querying it was a dashboard.

Point an agent at a database designed for human analysts and the workload changes shape as well as size. A load test that scales up requests per second will not show you the shape change.

Why agent traffic breaks a database built for dashboards

A human analyst asks one question, reads the result, thinks, then asks a better one. The pause between queries is load-bearing. It is what your connection pool was sized against and what your query cache was tuned for.

An agent has no pause. Handed something ambiguous like why margin dropped in one region, it introspects the schema, samples a few columns to see what the values actually look like, drafts several versions of the same join because it cannot tell which foreign key is the real one, runs all of them, and keeps whichever returned something plausible.

Generating more candidate queries raises the chance that one of them is right, so an agent that speculates less is an agent that answers worse. Deduplicating the traffic makes the feature weaker.

What makes this expensive is that your query planner treats each of those statements as a first-class request typed by a person. Ten concurrent agents each drafting eight variants of one join is eighty independent plans and eighty scans over the same fact table. Planners assume queries arrive independently, which held for as long as a person was doing the typing. The layer agent traffic wants is one that sees the batch: shared scans across sibling queries, caching at the sub-plan level rather than the statement level, and a latency estimate handed back to the agent before an expensive plan runs, so it can abandon the branch instead of paying for it.

What data infrastructure for AI workloads actually needs

The win here is not a new engine. It is narrowing what the agent is allowed to ask.

An agent given raw schema access behaves like a competent new hire with production credentials and no documentation. It guesses at joins and finds the soft-delete column late. It will also write syntactically perfect SQL against a deprecated table nothing has written to since the last migration and report the answer with total confidence. Hand it a small set of modeled tables where the metric definitions are already resolved, and most of the speculative fan-out has nothing left to speculate about. Revenue means one thing. There is one obvious table to ask, with no orphaned copy sitting beside it.

That is ordinary data engineering and analytics work — dbt models in version control, with a written definition and a freshness test behind every metric. It was worth doing before anyone mentioned agents. It becomes the difference between a working agent and a confident liar once they arrive.

The agent also needs a cost signal, because an autonomous system with no price feedback will happily scan the full event history to answer a question about yesterday. Its read path needs its own replica and its own budget, separate from the application’s, so a bad reasoning loop degrades a chat window rather than checkout. Whether that separate store is a warehouse or a lake is a genuine architectural decision with different failure modes on each side — we compare them in data warehouse vs data lake.

Give the agent a documented subset and a spend ceiling before you give it more capacity.

How to store memory for an AI agent

The default approach right now is a folder of markdown files: write down what was learned, grep or embed to retrieve it, paste the result into context. This works well for one session against one repository. It degrades once thousands of agents are working in parallel against a live warehouse, because retrieval by keyword or vector similarity cannot filter on the attributes that decide whether a memory is relevant — which module, which table, which kind of operation, which failure mode.

The sharper problem is what gets stored. Retrieval over raw agent history teaches new agents to repeat old errors. If the store holds the transcript of an agent trying the wrong join and failing, the next agent retrieves that transcript and tries the wrong join. Store conclusions instead: the corrected fact, tagged with the conditions under which it applies. Memory then needs a schema, with columns somebody chose on purpose. Decide those attributes for your own domain while the store is still small enough to migrate.

What breaks when an AI agent writes to your database

Databases already handle two users editing the same row at once. That machinery assumes each writer knows what it wants. An AI agent does not. It writes, observes the result, concludes the write was wrong, and issues a compensating change. Two agents doing that against the same rows can chase each other indefinitely, each one undoing the other’s correction. Nothing about that resembles a deadlock the database can detect and kill. It looks like healthy activity.

So agent writes need machinery the read path does without. Every agent-initiated write carries an idempotency key and the identity of the agent that issued it. State transitions go through a durable execution layer that can replay or compensate a half-finished sequence instead of leaving it half-finished. There is a hard ceiling on retries per task, enforced outside the agent, because an agent told to keep trying will keep trying. Add an append-only log recording which agent changed which row on what input, and you have the artifact you will need the first time an agent writes something a customer disputes.

Where to spend before you tune the model

Start by measuring. Log every query the agent issues alongside the request ID of the user question that produced it. That one change lets you compute fan-out per question and see which tables absorb the cost. Until that log exists, any capacity decision is a guess.

The remediation has a known shape. Getting from scattered sources to modeled tables with signed-off metric definitions and tested incremental loads is an indicative ₹6L – ₹9L engagement, the second of four tiers in our cost of data engineering guide. Weigh that against the prompt tuning and retry logic a team writes to work around a schema the agent cannot read reliably, none of which touches the schema.

Sometimes the honest answer is a set of curated views and a materialized rollup, with no platform at all. We would rather say that than sell a warehouse. Ask any vendor which of the two you need before they have seen your query log, then ask again after they have.

What happens if you ship the agent before the data layer

The database rarely falls over, and that is the trouble. It slows down, the pool saturates, somebody adds a read replica to relieve the pressure, and the agent can then answer from a replica that has fallen behind and report a stale number as a current one. Nothing alerts. A wrong number from an agent is indistinguishable from a right one, and the first person to notice is a customer reading it back to you.

The cost curve bends the wrong way too. Every additional agent multiplies against the same fact tables, so the bill grows in proportion to adoption. The infrastructure conversation therefore lands at the exact moment the feature is finally working, when nobody has any appetite for hearing that the foundation needs replacing.

Then there is the rework. Your prompts and your eval sets get written against whatever schema the agent found on day one. Change the schema afterwards and the agent gets rewritten alongside it, and the evals stop being comparable to the ones your team has already been tracking.

Pull last week’s query log and count how many statements one user question produced. If that log does not exist, you are running a workload you cannot see, and every month spent adding capacity instead of visibility makes the eventual rebuild bigger.