What Data Readiness for AI Actually Costs

The model layer is something you buy. Providers ship new ones every few months and they are close enough in capability for most business tasks. Swapping one for another costs you a config change and an evaluation run.

Three years of your own order history, joined correctly to your own customer records, has no vendor. It exists only if someone builds it. That asymmetry is why AI projects stall on data: teams budget for the part with a price list and meet the part without one halfway through. It usually goes the same way. A model gets wired to a few thousand sample rows, the answers look right, and the room agrees to fund it. Six weeks later the same model is reading live production data and nobody can explain why it thinks most of the customer base churned in March. The sample was clean because a person cleaned it by hand on a Tuesday afternoon. No such person exists inside the pipeline that has to run every night.

A support assistant grounded in your own documents needs those documents deduplicated, current versions only, each one carrying who is allowed to see it. Skip that and it answers confidently from a refund policy you retired eighteen months ago, and the first person to notice is a customer. Nothing in the prompt fixes it. Most of what gets called an AI build is data engineering with a chat window on the end.

Deciding which record is the customer

Identity comes first, and it is design work with real trade-offs. The same customer exists as a Stripe ID, a HubSpot record and a support email address. Until someone rules on which one wins, you cannot count customers, let alone train on them.

The ruling has edges. Somebody has to say what happens when the email matches but the company name doesn’t, and whether a customer who left and came back under a new billing entity is the same relationship continuing or a new one starting. Those answers change what a model sees. A churn model sitting on a rule that splits returning customers into fresh records will report churn that never happened, and it will report it consistently enough to look credible.

The rule belongs in code, where it runs on every load. Add a check that fails that load when a source starts emitting identifiers the rule can’t resolve, because the week a vendor changes their ID format is the week silent duplicates start accumulating.

Whether the history you assumed still exists

Week two is where teams find out that the two years of history the business case rested on was overwritten by a nightly job, or lives in monthly CSV exports whose column headings changed twice.

Operational systems are built to answer what is true now. Most of them update rows in place, so yesterday’s state of an order is gone unless something was capturing changes as they happened. A dashboard showing current pipeline runs happily on that. A model that has to learn what a deal looked like a month before it closed cannot, because the record has already been rewritten to its final state.

So the first real question in scoping is how far back your data is honest, and where that history physically sits. Pulling old data back through a metered API often takes longer than building the pipeline that keeps it current from now on, so the backfill needs its own end date in the plan. It gates everything downstream and cannot be parallelized past the vendor’s rate limit.

The metric argument engineering cannot settle

Definitions are where it gets political. Finance counts revenue on invoice date, sales counts it on close date, and each is right for its own purpose. A model trained across that disagreement learns the disagreement.

Nobody in engineering can arbitrate it. We can put both numbers on one screen and show what the gap between them costs; someone senior has to pick. The ruling then has to be written down where a new analyst will find it a year later, because an undocumented definition reverts to an argument the moment the person who made the call changes roles.

Its practical form is a single modeled table that everything else reads from. One place where revenue is defined, one place to change it. Reports built from ad-hoc queries against raw tables drift apart no matter how firmly the definition was agreed, because each query encodes the definition again and only one of them ever gets updated.

Lineage and access, before a feature makes it urgent

For every column: where it came from, and who is allowed to see it. Paperwork, until an AI feature surfaces one region’s data to another, or a subject access request arrives and you have to say which systems hold that person’s data.

Retrieval makes this sharper than reporting ever did. A dashboard shows a fixed set of fields to a fixed set of roles. An assistant with a document corpus will answer from whatever it can reach, so the permission has to travel with the document into the index and be enforced at query time rather than filtered out of the answer afterwards. Retrofitting that comes close to rebuilding the retrieval layer, which is why it belongs in the first design conversation and not the security review at the end.

Lineage earns its cost the first time a number is wrong in a meeting. Without it, tracing that number back to the source that broke means reading every transformation between the two by hand. With it, the same question is a lookup.

What a data foundation costs

Our cost of data engineering guide publishes the ranges we scope against. They are these.

A reporting starter sits at ₹1.5L–₹2.5L ($1.7k–$2.8k) and ships in three to five weeks: ingestion from a few well-documented sources, a warehouse, and dashboards answering questions you name up front. It buys you numbers you stop rebuilding by hand. It skips modeling and historical repair, so an AI project still starts from zero.

The tier most teams need before an AI project is a modeled warehouse. ₹6L–₹9L ($6.8k–$10k), eight to twelve weeks. Modeled tables, metric definitions signed off in writing, incremental loads, automated tests on the pipelines that matter. That is the point at which a retrieval corpus or a training set can be assembled without someone hand-fixing it each time, and the point at which the retrieval versus fine-tuning question becomes answerable.

Above that, ₹18L–₹28L ($20k–$32k) over four to six months gets orchestration with alerting and lineage, plus activation back into your CRM and ad platforms. Regulated or customer-facing analytics starts from ₹40L ($45k) and runs six to twelve months, because row-level security and audit evidence are a project in their own right.

Those are indicative ranges for a scoping conversation, not quotes. The figure that binds us comes out of a paid discovery against a written scope. What moves it most is the state of the data inside your source systems and how many of them there are. How far back you need history moves it next. Dashboard count barely registers by comparison.

The costs nobody budgets for

Historical backfill is the big one, and all of it lands before any of the value does. Dragging four years out of a rate-limited system has to finish before anyone can train anything, and it shows the business nothing while it runs.

Metric arbitration costs too, mostly in your calendar rather than ours. Two or three meetings where finance and sales settle on one definition of revenue, with someone in the room empowered to end the argument. Skip those meetings and you pay for them later, in front of a board being shown two different numbers for the same quarter.

Then the run cost, which never appears in a build quote because it starts the day the build ends. A managed warehouse bills for badly written queries, and a dashboard somebody refreshes every fifteen minutes can quietly cost more per month than the pipeline feeding it. Materialize the expensive aggregate once and let the dashboard read that.

Sources break, too. A vendor renames a field, an API version gets retired. That will happen, and what you are choosing is whether you hear about it from an alert or from a client. Freshness and row-count checks that fail the load rather than writing partial data are cheap during the build and awkward to retrofit. A half-loaded table does more damage than a missing one, because a missing table stops the report and a half-loaded table gets read.

When we’d tell you not to build one yet

Sometimes a warehouse would buy you complexity and a monthly bill and very little else, because everything already lives in one system whose built-in reports answer your questions. The warehouse versus lake decision only matters once you’ve established you need either.

If the source systems are being replaced in the next few months, wait. You would be modeling data that is about to change shape.

If nobody in the business has agreed what a customer or an order is, engineering can’t settle it for you. We can chair the conversation. We can’t decide it, and building on an unsettled definition produces a warehouse that is confidently wrong.

And if what you want is general visibility with no decision attached to it, you are asking for every table, which is a platform — the expensive version, and the one most likely to go unused.

Checking your own data before anyone quotes you

Most of the diagnostic you can run yourself, in an afternoon, with a spreadsheet:

• Name the one decision the AI is meant to change, and who currently makes it on instinct.
• Pull a real extract of a couple of hundred rows from each system that decision touches.
• Try the join by hand. Match the same customer across those extracts and count how many fail to line up.
• Ask two departments, separately, for the definition of the metric involved, then compare what comes back.
• Check whether those systems still hold the history your business case assumes, or only the current state of each record.

If the join holds and the definitions agree, your data is in better shape than most and the AI work can probably start. If more rows fail the join than pass it, and the two definitions come back different, you have just found what the next three months of work is, and it isn’t prompt engineering.

Start with the decision, because every other answer is downstream of it. Write down one recurring choice somebody in your business currently makes on instinct, and list the systems that choice touches. How many sources you need, how much history, which definitions have to be arbitrated, whether a warehouse is justified at all — every one of those falls out of those two lines, and no vendor can supply them for you. Arrive at a discovery call without them and assembling them becomes the first thing you pay somebody to do.