Verification Without An Oracle

The argument that verification becomes the binding constraint once AI collapses the cost of production is made elsewhere in this library, and it is made about software. This page is about what happens to that argument when you take away the thing software quietly relies on: an oracle.

An oracle, in the testing literature, is any mechanism that can tell you whether an output is correct without a human having to decide. A test suite is an oracle. So is a type checker, a compiler, a schema validator, a linter, a canary deployment that watches error rates, a diff against a known-good snapshot. None of them is complete and all of them can be wrong, but they share one property that matters enormously: they scale. Run ten times as much code through them and they cost you ten times almost nothing.

Now consider a team producing episodes of a serialised story, a weekly research publication, a campaign of social assets, or a documentary cut. AI has done to their production step exactly what it did to software's: an artefact that took a week takes an afternoon. And the question "is this any good" has no oracle at all. It cannot be asserted. It resolves to a person reading the thing, forming a judgement, and being right or wrong in ways that will not be known for months.

This is not a softer version of the software problem. It is a structurally worse one, and it is worth being precise about why.

Three properties a real oracle has, and taste does not

It is fast relative to production. A test suite runs in minutes against work that took days. Editorial judgement runs in roughly the time it takes to consume the artefact. You cannot read a script in less time than it takes to read a script. The ratio of verification time to production time was always poor here; AI made it catastrophic by shrinking only the denominator.

It is consistent. The same test on the same input gives the same answer today and next Thursday. Two editors given the same draft do not reliably agree, and one editor given the same draft on a Monday and a Friday does not reliably agree with himself. Verification with variance is not a gate; it is a lottery with a queue in front of it.

It parallelises. Add machines, verify more. This is the property people most expect to transfer and it transfers least. Adding reviewers to a judgement about coherence does not halve the time; it produces two opinions that must now be reconciled, which is a third piece of work performed by a more senior person who is scarcer than either. For anything that has to hang together — a season, a publication's voice, a brand — review is inherently centralising. There is a small number of people who can hold the whole thing in their head, and that number does not respond to hiring.

The arithmetic, stated plainly

Little's Law does not care that the work is creative. Cycle time equals work in progress divided by throughput, and throughput at a stage cannot exceed the capacity of that stage.

If production capacity rises tenfold and review capacity is unchanged, review becomes the constraint immediately and absolutely. Everything produced beyond review capacity does not become output. It becomes inventory: finished-looking artefacts sitting in a queue, ageing, going stale against a world that moves, and generating the specific misery of a team that can see a fortnight of completed work nobody has looked at.

Kingman's formula then does the rest. Queue time rises with utilisation, and it rises hyperbolically as utilisation approaches one. A single editor who is the only person who can sign off, and who is booked at ninety per cent, is not ten per cent away from fine. That person is the reason a two-day turnaround became three weeks, and no amount of production speed touches it.

Wait time against utilisation

The uncomfortable conclusion is that generating more than you can verify is not throughput. It is inventory, and inventory in a creative pipeline decays. A team celebrating that it now produces five times as much has not got faster unless the fifth artefact actually reaches an audience.

What to do instead of pretending review scales

Build the partial oracles you can actually have

You cannot automate taste. You can automate a surprising amount of what currently occupies the person exercising it.

Much of what a reviewer catches is not a judgement at all. It is a fact: a character described with the wrong eye colour, a figure that contradicts last quarter's, a name spelled two ways, a legal disclaimer missing, an aspect ratio wrong for the platform, a claim made without the source it needs. Every one of those is checkable mechanically, and every one of them that reaches a human is a unit of the scarcest capacity in the system spent on something a script could have done.

The move is to write down the facts your work must not violate — continuity, house style, factual and legal constraints, platform requirements — and check them automatically before anything reaches a person. This is the equivalent of a linter and a type checker, and it has the same effect: it does not decide whether the work is good, but it guarantees the reviewer is only ever spending attention on that question.

Put the WIP limit at the constraint, not the start

If review can absorb four pieces a week, produce four pieces a week. Producing twelve does not get you twelve; it gets you four, plus eight ageing in a queue, plus a team demoralised by the gap between effort and publication.

This is the hardest of these to accept, because production capacity now feels free and deliberately leaving it idle feels wasteful. It is not wasteful. The alternative is not more output; it is the same output with worse cycle time and a pile of rotting inventory. Idle capacity at a non-constraint is the normal, correct state of a balanced system.

Review earlier, smaller, and at the level where the decision is cheap

The expensive failure is a finished artefact rejected on premise. The cheapest oracle available in creative work is a human being looking at an outline, a rough, a thumbnail or a first two hundred words — before production cost has been incurred, and at a size where judgement takes minutes rather than hours.

AI has made this strictly better, not worse, because producing three cheap variants to choose between at the outline stage now costs almost nothing. The failure mode is using that same capability to produce three finished variants and asking the constraint to choose between them, which triples the load on precisely the stage that has none to spare.

Give coherence a single owner and everything else a distributed one

Split the judgement. Anything that is locally checkable — does this paragraph work, is this shot clean, is this claim supported — distributes fine across many reviewers. Anything that concerns whether the piece fits the whole does not, and pretending otherwise produces committee drift that nobody will own and everyone will resent.

Name the person who holds coherence. Protect their time as the constraint it is. Measure their queue, because it is the system's queue.

What this says back to software

Software teams reading this should not feel comfortable. The oracle they rely on is narrower than it looks: tests verify what someone thought to assert, and the question "is this the right change" has never had an oracle in software either. What AI changed is that the volume of changes arriving at that unautomatable judgement went up by an order of magnitude while the number of people qualified to make it stayed flat.

Creative production is the case where the problem is undisguised, because nobody there was ever able to pretend a green build meant the work was good. The discipline it forces — write down what can be checked, check it mechanically, protect the judgement that cannot be, and never produce faster than you can accept — is the same discipline software now needs and can still avoid noticing it needs.

What to do on Monday

1. Measure the ratio. For the last ten things you published, record the time from "production finished" to "approved". If that exceeds the time spent producing, verification is your constraint and it has been for a while.
2. Count your queue. How many finished artefacts are waiting for review right now? That number is inventory. Say it out loud in those terms.
3. List the last thirty review comments and mark each one fact or judgement. The fact ratio is usually somewhere above half, and every one of them is automatable.
4. Build one check this week from that list — the dullest, most mechanical one. Continuity of names, or required disclaimers, or asset dimensions.
5. Find out the real utilisation of whoever signs off. If it is above eighty per cent, stop discussing production speed until that is fixed, because nothing upstream of a saturated constraint can improve the outcome.
6. Set a production cap equal to review capacity, and hold it for one month. Then compare published output, not produced output, against the month before.