Would a scored baseline on 50 of your service emails be useful before the Dynamics 365 move?
By whenNo deadline. A one-line reply to jon@vocabotics.com is plenty.
1What this shows
For each customer email to the service or sales desk, it sorts the message (breakdown, planned service, spares, new equipment, accounts, technical advice) and pulls out the machine, even when the customer calls it a TCU or a sprue picker, and any job or part number. It marks it urgent when a press or line is down or there is a safety risk. It holds safety risks, repeat failures, invoice disputes, warranty claims and large new-equipment enquiries for a person. It drafts the job sheet or reply for approval.
10/10
type right on 10 emails it had not seen (set 2)
3/4
messages needing a person were caught
9/10
urgency right on the same 10
0
sent to a person unnecessarily
It is one reusable engine, set up by two small files: the categories in plain English, and the rules. Every figure here is from the rules, which anyone can reproduce on a laptop in a second. A Claude version of the same engine (claude-sonnet-5-5) has not been run.
2Where it goes wrong
Each held-out set was written separately, from the category definitions only, and scored once with the rules frozen. Set 1 scored 13 of 15 on type but missed the case that mattered most: operators wedging a guard-door switch open to jog a robot, which should always go to a person. I widened the safety wording and a fresh set scored 10 of 10 on type and 3 of 4 on needing a person.
The miss in the fresh set was a disputed invoice ("we won't be paying this as it stands"), which went through as an ordinary accounts message. Keyword rules catch phrases, not intent, so this is the bar the Claude version has to beat on the same sheets before it replaces them.
3What made-up data cannot show
- A real inbox. Both held-out sets were synthetic and written by someone who knew the categories. Real customers write messier emails.
- Your machines and parts. The machine names, job numbers and parts lookups here are invented.
- Your approval rules. Nothing is dispatched or quoted without a person approving. Which cases need one is yours to set.
What a real version would take
About 200 past service and sales emails, redacted and labelled once, become the test sheet. Approval stays with people: nothing is dispatched or quoted without one clicking approve. The category, machine and job sheet land as a case or service order in Dynamics 365, so the classifier can be built now and moved across with the migration.
AEvidence
Source: results/accuracy_rules_holdout2.md, results/accuracy_rules_holdout.md and results/accuracy_rules_cases.md. Rules written against the 20-case sheet, then frozen for each held-out set
| Measure | 20-case sheet (tuned on) | Held-out set 1 (15, earlier rules) | Held-out set 2 (10, current rules) |
|---|---|---|---|
| Type (category) correct | 20/20 | 13/15 | 10/10 |
| Machine type | 20/20 | 14/15 | 10/10 |
| Job or part number | 20/20 | 15/15 | 10/10 |
| Urgency correct | 20/20 | 14/15 | 9/10 |
| Messages needing a person, caught | 5/5 | 4/5 | 3/4 |
| Sent to a person unnecessarily | 0 | 2 | 0 |
Source: results/results_rules_holdout2.csv
| # | Message (synthetic) | Type | Urgency | Handled by |
|---|---|---|---|---|
| SJ01 | Chiller on line 3 keeps tripping on... | Breakdown | high | Agent |
| SJ02 | This is the third time the dryer on... | Breakdown | high | Person |
| SJ03 | Hi, can you give me price and lead time... | Spares | normal | Agent |
| SJ04 | Morning, chasing our PO 88213 -... | Spares | normal | Agent |
| SJ05 | Our two process chillers are due their... | Planned service | normal | Agent |
| SJ06 | We are fitting out a new production... | New equipment | normal | Person |
| SJ07 | do you hire out TCUs? need one 90... | New equipment | normal | Agent |
| SJ08 | Hello accounts, invoice 55102 for job... | Accounts | high | Agent |
| SJ09 | Quick one - what drying temp and time... | Technical advice | normal | Agent |
| SJ10 | URGENT. Burning smell coming from the... | Breakdown | high | Person |
Source: results/results_rules_holdout2.csv
This is the third time the dryer on press 7 has failed since your engineer was here under job SJ-20419. Dew point won't come down and we have had to stop the press again. I want this escalated to your service manager today, we can't keep losing production like this.
What it did. Breakdown, urgency high. Held for a person because it is a complaint about a repeat failure. The draft is a job sheet for approval with a parts lookup slot, marked for a colleague to review before it is sent.
Source: this note
Source: not run. It would have to beat these figures on the same sheets
Next
- Reply with one line if a scored test on your own messages would be useful. No form, no commitment.
- Send 50 past service emails, redacted. I run the same sketch and send back the scored sheet, so the classifier starts from a measured baseline.
- Or start with a review. The Opportunity Review is
£1,950fixed over one week, remotely: a 60-minute call, a short questionnaire and up to three sample documents. You receive a written review of about eight pages. It is paid at booking and credited in full against a Discovery Sprint booked within 30 days.
Jon Ross
VOCABOTICS LTD