Skip to main content

Method · Tests agreed in week 1

We agree how you will check the AI before we build it.

Every build starts with acceptance tests written in plain English, in your own numbers, and agreed in the first week. The final milestone is paid when they pass, not when we say so.

Why tests first

Because “it seems to work” is not a result.

  • AI is right most of the time

    That is the problem. A demo that works on five examples says little about the five hundredth. Only a test set from your own work shows how often it is right, and how it fails.

  • “Done” has to be written down

    Without written tests, done means whatever the supplier says. With them, done means a number you agreed before any build work started.

  • It catches bad ideas early

    If the tests cannot be written, the job is not clear enough to build yet. Better to learn that in week 1 than in week 6.

Week 1

Four steps, five working days.

  1. Day 1–2

    Collect real cases

    You share past examples of the job, with the answer your team gave: last quarter's invoices and which were rejected, past questions and the right passage. Redacted is fine.

  2. Day 2–3

    Draft the tests

    We write each test in plain English, in your numbers: what goes in, what must come out, and how often. Each one names who checks it.

  3. Day 3–4

    Agree the bar

    You edit them. Some rules are absolute (it never approves a payment on its own); some are rates (it flags at least 95% of the bad ones). You decide which.

  4. Day 5

    Sign them off

    The agreed tests go into the contract as the acceptance criteria. From then on, every milestone report shows the latest scores against them.

A template you can use

What a test plan looks like.

An illustration with example numbers, not a client's plan. Yours uses your own cases and your own bar. Copy it freely.

TestKindGivenIt mustChecked how
T1RateGiven 200 past invoices your team rejected or approvedFlags at least 95% of the rejected onesRun by us, checked by you
T2NeverAny invoice over your approval limitNever approved without a named personRun on every release
T3RateGiven 100 past staff questions with a known answerAnswers at least 90% correctly, each with its source passageSpot-checked by your expert
T4NeverA question the documents do not answerSays it does not know, and does not guessRun on every release
T5TimeA normal working day's volumeEach item processed within 2 minutesMeasured live in week 6

During the build

Scores at every milestone.

  • The tests run on every change, so a fix in one place cannot quietly break another.
  • Each milestone report shows every test, its latest score and what changed.
  • If a test turns out to be wrong, changing it is a written change request, agreed by you.
  • At handover the test suite is yours, so you can keep checking after we have gone.

The rules we keep

Where the line sits.

  • A test that fails is reported as failed. We do not round up.
  • Where the AI decides something that matters, a person approves it. That is a test too.
  • If the tests show the idea will not reach the bar, we say so, early.

See it on your own job

Reply “sketch”, and get three draft tests for free.

Email a paragraph about the job to jon@vocabotics.com with the word sketch. You get one page back: three draft tests, the biggest risk and a price band.

We run our own company on the same habits. Read the Console case study.

    We use cookies.

    Tests agreed in week 1: how we decide an AI build is done · vocabotics