Document Processing Automation: Where Invoice Agents Fail

A document processing automation pipeline reads a scanned supplier invoice, lifts the vendor name, the invoice number, the line items and the total, and writes a payable into the ERP. On this particular page the total’s leading digit sits half across a fold, and the OCR pass reads it as something else. The agent has no way to know that. It carries the figure forward, fails to match the amount against the purchase order, treats the mismatch as a bad PO reference, goes looking for a better candidate, finds a plausible one from the same vendor, and files a clean payable against the wrong contract. Nothing throws. The dashboard shows one document processed.

The documents an agent gives up on are cheap. They land in a queue and a person opens them. What costs you money is the document it handles confidently and gets wrong, because from the outside a bad extraction looks like a good one. Same shape, same fields populated, same green tick.

Why document processing automation fails silently

Extraction sits at the front of a chain. Everything downstream — matching, approval routing, posting, payment — consumes what came out of that first step and has no independent view of whether it was right. A digit that shifts early doesn’t produce a smaller error later. It changes which branch the whole run takes.

That is what separates document work from most agent work. A support agent that misunderstands a customer produces a bad reply, and the customer says so. An extraction agent that misreads a field produces a well-formed record that moves through every system downstream before anyone with context sees it, and by then the context is gone. Someone is looking at a payment that already cleared, working backwards to figure out which vendor it should have gone to.

So the design problem moves to the front of the pipeline. It has to decide, field by field, whether it has evidence for the value it just produced from some source other than the model that produced it.

Why scanned and legacy documents break extraction

Clean documents get solved on day one. Native PDFs with a text layer, generated by the same accounting system every month, sit close to solved and always have — a template and a few patterns handled those long before anyone attached a model to them.

The remainder is where the work lives. Photographs of paper taken at an angle. Faxes rescanned twice. Tables that break across a page boundary so the header row is orphaned from half its rows. Stamps and handwritten approvals sitting over printed figures. Vendors who put the tax on a separate page. Decade-old contracts where the amendment that matters is a scanned insert in a different font.

Databricks built a benchmark for that material, OfficeQA Pro, which tests parsing and grounded reasoning over scanned and legacy files. The first model to clear 50% accuracy on it was GPT-5.5, in Databricks’ announcement of May 15, 2026.

That benchmark scores document question-answering in an agent harness rather than field-level extraction, so read it as a statement about the document class and not as an error rate for your invoices. This material is unsolved at the frontier. Nothing downstream in your pipeline will flag a wrong field unless you build the flag.

Checks that catch a wrong number before it moves money

A model’s opinion of its own output is weak evidence. Arithmetic and cross-referencing are strong evidence, and both are cheap to run on every document.

For invoices, the checks that earn their place are the ones a good accounts clerk performs by reflex. Line items sum to the subtotal. Subtotal plus tax equals the total. The tax figure matches the rate for that vendor’s jurisdiction. The invoice number has never been seen before from that vendor. The total sits within tolerance of the open purchase order, and the bank details match the ones on file, which is the check that catches a redirect printed onto the invoice itself.

The checks only need the extraction structured well enough to test: a typed schema with named fields, in place of a paragraph of prose the agent wrote about the invoice. So the schema gets designed before the prompt does, and a large share of the build effort goes into the integration and API layer holding the vendor master, the PO records and the duplicate index that the checks run against.

Where arithmetic doesn’t apply — contracts, clinical notes — the equivalent is provenance. Every extracted field carries the page and the region it came from, so a reviewer who doubts a value clicks it and sees the pixels. Capture that region at extraction time. Reconstructing it afterwards means re-running the document.

What to do with a model’s confidence score

A confidence score is easy to over-trust. It reports how typical the output looks, which is a separate question from whether the output matches the page. A model can be entirely confident about a smudged digit.

Feed the score into a routing decision alongside the hard checks. A field with high confidence that fails the arithmetic goes to a human. A field with low confidence that passes every cross-reference can go through untouched. Where confidence earns its keep is in aggregate: a template change shows up first as drift in the average for one vendor’s documents, and you want to know on the day it happens instead of at month end.

Track it per document type and per source, and alert on the change.

Which documents should go to a person

Every pipeline needs an exception queue, and the honest target for its size is well above zero.

Route on properties of the failure. Anything that fails a hard check. Anything above a value threshold the business sets — a payable over a certain size gets human eyes however clean the extraction looked, because the money at risk justifies the review on its own, whatever the model reported. Any document type absent from the evaluation set. Anything where a field the schema marks required came back empty.

Then measure the queue, because that number tells you whether the automation is working. If most documents route to review, you have a slower version of manual entry with a token bill attached. If nothing routes to review, the checks aren’t checking. What you want is a queue small enough that the people working it read each item properly, plus a sampling process that pulls a share of the auto-approved documents for review anyway. That sample is how the pipeline’s decay becomes visible to you before it becomes visible to your auditor.

Build the reviewer’s screen properly: extracted fields on one side, the source page on the other, edit in place, every correction written back to a store the pipeline can query later. Those corrections are what a monthly accuracy figure gets calculated from, and they’re the training data for whatever you decide to tune next.

Should you fine-tune a model on your own documents?

Usually no, and the case gets weaker with every frontier release. The argument for fine-tuning was that base models couldn’t read your particular species of ugly document, so you’d fund a labeling program and a training run to get one that could. Each release absorbs more of that capability, and a training run already paid for now competes against a base model that improved for free.

What never gets absorbed is everything specific to your business: your vendor master, your PO matching rules, your approval thresholds, the fact that one supplier bills in a currency their contract doesn’t mention. That knowledge belongs in retrieval and in the checks, where you can read and change it without a training cycle. The trade-off between retrieval and fine-tuning is worth settling before anyone budgets for labeling.

Providers also change model behavior without asking, and on document work the change surfaces as slightly different extractions rather than as failures — a date format that flips, a field that starts including the currency symbol. So the pipeline needs a fixed evaluation set: real documents with known-correct extractions, scored before launch, re-run after every model change, and diffed field by field.

What a document pipeline costs to build

Document variety moves the price more than anything else. One vendor’s invoice template is a different project from your entire supplier base, and the second is mostly a long tail of edge cases you can’t enumerate up front.

Write access moves it next. An agent that posts to a system of record pulls approval gates and a reversal path into scope; one that only proposes, and leaves a person to accept, does not. Then reference data. Checks that need a vendor master you don’t have yet mean building the vendor master first, and that work belongs in the estimate.

A first build scoped to one document type from one source, proposing rather than posting, with a review queue behind the checks, is contained work. Once it writes into the ERP itself it moves up a tier — approval gates, tracing, an evaluation set and a staged rollout — and our scoping guide for agent builds sets out where each tier lands and what each one excludes.

And if the documents are genuinely uniform and no step needs judgment, say so early. That is an AI automation build: a deterministic pipeline where the model handles extraction and fixed logic decides everything after it. Cheaper and more predictable than an agent that picks its own next move, and we would rather scope it that way.

Where to start

Pick the highest-volume document type you have. Pull a batch of real examples, weighted toward the ugly ones nobody wants to open, and extract them into the schema you think you want. By hand. That exercise answers the questions the whole build turns on: which fields are genuinely always present, and which checks would have caught the errors you just found. Then build for that one type and run it alongside the manual process. Compare the two on the documents where they disagree, because those disagreements are the only accuracy number worth quoting to a finance director. If you want a second pair of eyes on the schema before you commit to it, that is what the first week of an AI agent engagement goes on.