Your AI reviewer needs a light test

Three checks, one blind spot

On the decommissioning programme behind The £6M Thread Pull, every link was verified three ways before anything was switched off: traffic monitoring, inventory records, and confirmation from the service owner. It was a rigorous process, and it worked on thousands of circuits.

Then one circuit failed a migration twice. It had been running for over a decade and wasn’t in any database. All three checks assumed the circuit was recorded somewhere, so all three were blind to it. We fixed it by tracing the fibre ourselves between two buildings, and we added a physical end-to-end light test as a standard step from then on. That’s The Light Test.

The lesson I took from it: three checks that rely on the same source aren’t three checks. They’re one check, repeated.

The same trap in AI work

AI evaluation falls into this trap easily. A model’s output gets checked by another model, against test cases written from the same assumptions, by someone who already expects it to work. Each step looks like verification. None of them touches the thing the output is meant to describe.

The light test was valuable because it didn’t ask any record what was true. It sent light down the fibre and looked at what came out.

What that looked like in Redline

When I built Redline, a tool that reviews freelance contracts before they’re signed, I designed it around a check a person can do without trusting the tool: every flag quotes the exact clause it came from. A reviewer reads the quote, finds it in the contract, and decides for themselves. If Redline can’t point to the text, it doesn’t raise the flag.

To test it, I didn’t ask whether the output looked sensible. I planted six risky clauses in a sample contract, so I knew the right answer before I started. Redline caught five, each with the correct severity and an exact citation.

That’s one run, not a measured accuracy rate, and I say so on the case study. But it was checked against something the tool didn’t write.

The question to ask

For any AI system going into real work, I’d ask the same question I’d ask of a migration plan: which of our checks would still catch a problem if every record we have is wrong? If the answer is none of them, the programme needs a light test.

All three verification methods trusted a record that didn’t exist. The fix was standing at the exchange, not another database check.