Method · Tests agreed in week 1
We agree how you will check the AI before we build it.
Every build starts with acceptance tests written in plain English, in your own numbers, and agreed in the first week. The final milestone is paid when they pass, not when we say so.
Why tests first
Because “it seems to work” is not a result.
AI is right most of the time
That is the problem. A demo that works on five examples says little about the five hundredth. Only a test set from your own work shows how often it is right, and how it fails.
“Done” has to be written down
Without written tests, done means whatever the supplier says. With them, done means a number you agreed before any build work started.
It catches bad ideas early
If the tests cannot be written, the job is not clear enough to build yet. Better to learn that in week 1 than in week 6.
Week 1
Four steps, five working days.
- Day 1–2
Collect real cases
You share past examples of the job, with the answer your team gave: last quarter's invoices and which were rejected, past questions and the right passage. Redacted is fine.
- Day 2–3
Draft the tests
We write each test in plain English, in your numbers: what goes in, what must come out, and how often. Each one names who checks it.
- Day 3–4
Agree the bar
You edit them. Some rules are absolute (it never approves a payment on its own); some are rates (it flags at least 95% of the bad ones). You decide which.
- Day 5
Sign them off
The agreed tests go into the contract as the acceptance criteria. From then on, every milestone report shows the latest scores against them.
A template you can use
What a test plan looks like.
An illustration with example numbers, not a client's plan. Yours uses your own cases and your own bar. Copy it freely.
| Test | Kind | Given | It must | Checked how |
|---|---|---|---|---|
| T1 | Rate | Given 200 past invoices your team rejected or approved | Flags at least 95% of the rejected ones | Run by us, checked by you |
| T2 | Never | Any invoice over your approval limit | Never approved without a named person | Run on every release |
| T3 | Rate | Given 100 past staff questions with a known answer | Answers at least 90% correctly, each with its source passage | Spot-checked by your expert |
| T4 | Never | A question the documents do not answer | Says it does not know, and does not guess | Run on every release |
| T5 | Time | A normal working day's volume | Each item processed within 2 minutes | Measured live in week 6 |
During the build
Scores at every milestone.
- The tests run on every change, so a fix in one place cannot quietly break another.
- Each milestone report shows every test, its latest score and what changed.
- If a test turns out to be wrong, changing it is a written change request, agreed by you.
- At handover the test suite is yours, so you can keep checking after we have gone.
The rules we keep
Where the line sits.
- A test that fails is reported as failed. We do not round up.
- Where the AI decides something that matters, a person approves it. That is a test too.
- If the tests show the idea will not reach the bar, we say so, early.
See it on your own job
Reply “sketch”, and get three draft tests for free.
Email a paragraph about the job to jon@vocabotics.com with the word sketch. You get one page back: three draft tests, the biggest risk and a price band.
We run our own company on the same habits. Read the Console case study.