Right nine times out of ten is wrong four times a day

A demo runs once. Somebody hands a model an invoice, it reads the invoice, it finds the purchase order, it says the two match. Everybody in the room nods, and they are right to: it is impressive.

Then the process goes live, and it does not run once. An accounts team matching invoices runs the same job forty times a day. At ninety per cent, that is four invoices a day that are wrong — not flagged, not paused, just wrong, with the same confident tone as the thirty-six that were right.

The problem is not the model

Ninety per cent is a good score for a model reading a messy document. The problem is asking one model to do the whole job: read the invoice, find the order, decide what counts as a match, decide what to do about a mismatch, and post the result. Every one of those is a place for it to find a new way of doing things, and it will.

Fine in a demo. Not on an order desk with eleven steps and a customer on the phone.

So we split it

The way we build is simple to say. The path is explicit logic and it does not move. Fetching the invoice, looking up the order, comparing the totals, posting to the ledger — these are steps a machine can do the same way every time, so they are written that way.

Judgment is kept to one step. Reading a supplier's free-text note, deciding whether a line description means the same thing as the one on the order — that is where a model earns its place, and it is the only place it gets.

Where the two meet, you put a gate: a rule about what the judgment step is allowed to pass through on its own, and what goes to a person.

You can predict what it will do tomorrow and check what it did yesterday.

That sentence is the test. If a workflow cannot pass it, it is not ready to run while nobody is watching.

What this buys you

• Predictable failures. When something goes wrong, it goes wrong at a known step, for a reason you can read.
• A shorter review queue. People only see the cases the gate sends them, not every run.
• Room to improve. You can swap the model in the judgment step without touching the path around it.

The north star is not an agent that can do anything. It is a process your team stops thinking about, because it does the same thing every day and tells you when it cannot.