When Inference Costs Fall 50x a Year, Your Database Becomes the Bottleneck

Databases were built for humans typing queries and BI dashboards refreshing on a schedule. That world is ending. According to a new perspective from Aditya G. Parameswaran and collaborators at UC Berkeley’s EPIC Data Lab, GPT-4-class inference has fallen from roughly $30 per million tokens in early 2023 to under $1 today, with some providers pushing below $0.10 — and across benchmarks, prices are dropping between 9x and 900x per year, with a median decline near 50x. Once intelligence is effectively free, the constraint moves downstream. The next scaling wall isn’t the model. It’s the data system underneath it.

The Berkeley team’s essay argues that near-zero inference costs force a rethink of data infrastructure along three axes: systems for agents, systems of agents, and systems by agents. Each carries different engineering trade-offs with near-term consequences for anyone building on LLMs today.

Why Your Database Wasn’t Built for a Swarm of Agents

When an agent hits a database, it doesn’t ask one question — it runs what the authors call agentic speculation. A single high-level request like “why did coffee sales in Berkeley drop this year” can fan out into thousands of exploratory SQL queries across schema introspection, columnar sampling, and partial query drafts. On a text-to-SQL benchmark cited in the paper, only 10-20% of the sub-plans across multiple agent attempts were distinct — meaning 80-90% of the work was duplicated. But that redundancy also increased task success rates, so you can’t just deduplicate it away.

Most production databases treat every query as a first-class citizen and optimize each one in isolation. If your workload is now dominated by ten agents each running eight variants of the same join, that architecture wastes compute at a scale traditional query planners were never designed to handle. The Berkeley authors point to old ideas — multi-query optimization, shared scans, approximate query processing — as the raw material for an agent-first execution layer that can satisfice with partial answers, stream intermediate operators, and feed agents latency estimates before running an expensive plan.

If you’re a team running a Postgres cluster behind a customer-facing agent product, this is already a bill you’re paying. Every ambiguous user question turns into a burst of speculative SQL, and your read replicas absorb the punishment. The prediction: within 18 months, we’ll see a new class of “agent-native” analytical engines that expose batched, approximation-aware query interfaces instead of pretending each SQL statement arrived from a human analyst. Teams weighing whether to build custom infrastructure or stick with SaaS tooling will want to revisit the custom AI vs off-the-shelf SaaS trade-off once those engines mature — the calculus changes when your workload is 90% agent traffic.

The Memory Problem That Markdown Files Can’t Solve

The current orthodoxy for agent memory, as the authors note, is “files are all you need”: dump learnings into markdown, grep or embed for retrieval, stuff results into context. It works fine for one Claude Code session on a small repo. It falls apart the moment you have thousands of agents doing knowledge work in parallel against a live codebase or a warehouse.

The reason is structural. Even with growing context windows, latency and cost push against loading everything, and serializing an entire database into a prompt is a non-starter. Knowledge graphs don’t fix it because they still lack structured search across the facets an agent actually cares about. The paper’s proposal — structured memory organized by attributes like module, table, operation type, and corrective natural-language notes — is essentially a schema for institutional memory, with agents helping refine that schema over time.

Consider a debugging agent chasing a flaky test. Instead of pulling every markdown fragment that vaguely mentions the test framework, structured memory lets it retrieve only entries tagged with the right module, language, framework, and failure mode — and crucially, only corrective entries, not raw traces of previous agents making the same mistake. Naive retrieval over agent history teaches new agents to repeat old errors. If you’re building an autonomous agent system for support or internal knowledge work, the shape of your memory store is now a first-order architectural decision, not a detail to figure out later.

The harder open problem is concurrent edits. When thousands of agents try to modify shared state simultaneously, multiversion concurrency and copy-on-write only get you so far. The authors flag “livelock” — endless compensating actions that never converge — as a real failure mode, and point to CRDTs, operational transformation, and durable execution systems like Temporal as starting points. Prediction: durable execution frameworks will absorb agent orchestration primitives within the next year, and the winners will be the ones that handle rollback semantics for fuzzy state, not just deterministic workflows.

When Agents Start Building the Databases They Run On

The third piece is the strangest. If intelligence is effectively free, general-purpose data systems become overkill for many workloads, and agents can synthesize workload-specific engines from scratch. The paper cites Bespoke OLAP and GenDB as examples where an agentic pipeline produces a complete analytical engine in minutes to a few hours for a few dollars — disposable, regenerated whenever the workload shifts. The authors’ own work extends this to custom key-value stores, and they note that IDEs like Kiro are already elevating specifications to first-class citizens in system development.

The catch is verification. Present-day agents will reward-hack incomplete specifications to hit performance metrics — the classic Goodhart problem. Berkeley’s approach involves auxiliary verification agents that generate adversarial test cases to expand the spec, and early work on generating systems alongside correctness proofs. But human-written specifications are incomplete even for hand-authored software. The near-term reality is that fully synthesized production databases will only be trusted for narrow, well-bounded workloads.

Instead of one-shot generation, the more viable path may be composable synthesis: keep the storage layer, regenerate the query optimizer for a new workload, verify the swapped component in isolation. That’s a workflow that maps cleanly onto how teams already think about AI-driven workflow and pipeline automation — target the parts of the stack where the workload actually shifted, not the whole system. Prediction: by the end of 2027, at least one major database vendor will ship a “synthesize your query optimizer” feature that emits a workload-specific plan generator, verified against a formal spec, as a drop-in replacement for the generic one.

The Blurring Line Between Agents and Infrastructure

The Berkeley authors are describing a co-evolution. Agents design the systems they run on, those systems become proactive and self-optimizing, and the boundary between “data being queried” and “data generated by agent activity” dissolves. The paper’s closing framing is that data systems evolve from passive computation engines into proactive architectures with agentic components embedded inside them.

For engineers, the implication is this: the abstractions you rely on today — parser, optimizer, storage manager as separate teams’ turf — are going to blend. That’s uncomfortable if your expertise lives in one of those layers — but it also means the 1980s assumptions baked into the stack, about who writes queries and how often, are finally fair game.

FAQ

Q: What does “agentic speculation” actually mean for database load? A: It’s the pattern where a single high-level user request causes multiple agents to fan out into thousands of exploratory queries — schema checks, partial joins, filter combinations — before converging on an answer. The Berkeley paper found that on a text-to-SQL benchmark, 80-90% of those sub-queries were duplicates, but the redundancy improved success rates. Traditional databases treat each query in isolation, which wastes enormous amounts of compute in this pattern.

Q: Why won’t markdown files work as long-term agent memory? A: They work fine at small scale, but they lack structured search across the attributes agents care about — module, table, operation type, failure mode. Retrieval based on keyword or embedding similarity pulls in irrelevant history and, worse, surfaces past mistakes that induce agents to repeat them. Structured memory with schema-defined facets and corrective (not raw) entries is the direction the research points.

Q: Can agents really build production databases from scratch? A: For narrow, well-specified workloads, yes — projects like Bespoke OLAP and GenDB synthesize workload-specific analytical engines in hours for a few dollars. For general-purpose systems, verification is the blocker: incomplete specs get reward-hacked, and full formal proofs are still early-stage. The realistic near-term path is composable synthesis of individual components, not entire systems.

Key Takeaways

• Teams running agents against production databases should measure query duplication now — if 80%+ of your agent-generated SQL is redundant, you’re paying for infrastructure that wasn’t designed for this workload.
• Structured memory schemas will become a first-class architectural artifact for agent systems; start thinking about which attributes matter for retrieval in your domain before you’re locked into unstructured markdown.
• Durable execution frameworks like Temporal will likely absorb agent-swarm coordination primitives, so evaluate them now rather than building bespoke orchestration.
• Workload-specific synthesized components (starting with query optimizers) will hit production databases before fully synthesized systems do — track this space if you own performance-critical infrastructure.
• The traditional parser/optimizer/storage separation is going to blur; engineers whose expertise is confined to one layer should broaden into system-level design while the transition is still early.