Testing Non-Deterministic Systems

The first test an engineer writes against a language model is almost always an equality assertion. Given this input, expect this string. It passes on the machine where it was written, fails on the next run, and is deleted within a fortnight. The engineer concludes that these systems cannot be tested, and the team proceeds on vibes and demos.

That conclusion is wrong, but the observation behind it is correct. Assertion-based testing rests on an assumption that no longer holds: that for a given input there is exactly one correct output, and any deviation from it is a defect. A model-backed component produces a sample from a distribution over plausible outputs. Several of those outputs are equally correct. Many more are acceptable. Some are wrong. Equality cannot tell these apart, and a test that fails on a correct answer is worse than no test, because it trains people to ignore red.

What replaces it is not a single technique but a redistribution of effort. Most of the system around the model is entirely deterministic and should be tested exactly as it always was, with more rigour rather than less. The model-shaped hole in the middle gets a different class of instrument: properties and invariants that must hold for any acceptable output, distributional checks over many samples, and a scored evaluation harness that treats quality as a measurement rather than a boolean.

Getting the split right is most of the work. Teams that fail usually fail by treating the whole system as untestable because one component is probabilistic.

Draw the deterministic boundary first

Before deciding how to test the uncertain part, be precise about how small it is.

Take a typical assistant feature. A request arrives and is authenticated. Input is validated and normalised. Retrieval runs against an index. Context is assembled and truncated to a budget. A prompt is rendered from a template. A model is called, with retries and a timeout. The response is parsed, often as structured output. It is validated against a schema. Tools are dispatched based on it. Results are persisted. Something is rendered.

Exactly one step in that list is non-deterministic. Everything else is ordinary software with ordinary failure modes, and in practice most production incidents in these systems come from those steps rather than from the model: a truncation bug that silently drops the most relevant retrieved chunk, a parser that throws on a response wrapped in a code fence, a retry that duplicates a side effect, a template that renders an empty string when a variable is missing.

Test all of it conventionally and aggressively. Prompt rendering is a pure function of a template and variables; assert on the rendered string. Context assembly and truncation are pure functions; assert on what survives when the budget is exceeded. Parsers should be tested against a library of real malformed responses collected from production. Schema validation, tool dispatch, retry and timeout behaviour, and error paths are all deterministic, and they are where cheap confidence lives.

Control what you can, then stop pretending

There are two knobs that reduce variance, and both are useful and neither is sufficient.

Temperature and sampling parameters. Setting temperature to zero makes the model pick the highest-probability token at each step. This substantially reduces variation and is the right default for tests. It does not guarantee identical output: batching, hardware, floating-point non-associativity and provider-side changes all introduce variation, and greedy decoding can flip between two near-equal candidates on a trivial numerical difference.

Seeds. Where a provider exposes a seed parameter, it makes sampling reproducible for a fixed model version and configuration. That qualifier does a lot of work. Seeds do not survive a model update, and models update.

Use both in tests, because reducing noise makes real signal easier to see. Then design your tests so they would still be valid if both were unavailable, because on the day the provider changes something underneath you, they effectively will be.

One thing worth stating plainly: do not run your evaluation at temperature zero and then serve production at a higher temperature. You will have measured a system you are not running. If production samples, your evaluation should sample too, and should run enough samples per case to characterise the spread.

Properties and invariants instead of equality

The productive move is to stop asserting what the output is and start asserting what must be true of it. Borrowed from property-based testing, this works unusually well here because most real requirements are properties rather than exact strings.

Structural properties. The response parses as valid JSON. It validates against the schema. Required fields are present. Enumerated fields contain only permitted values. This is the cheapest and highest-value category by a wide margin, and it should be enforced at runtime as well as in tests.

Grounding properties. Every factual claim is attributable to a retrieved source. No citation refers to a document that was not in the context. Numbers appearing in the answer appear in the source material. For a retrieval-backed system these are the properties that catch the failure you care most about.

Safety and policy properties. The output contains no content matching prohibited categories. It does not reveal system instructions. It does not include personal data that was not in the input. Refusal behaviour holds on inputs that should be refused.

Consistency properties. Semantically equivalent inputs produce semantically equivalent outputs. Reordering items that should not affect the result does not change it. The same question asked twice does not produce contradictory answers. These are metamorphic relations — you assert a relationship between two runs rather than a fact about one — and they are the sharpest tool available for a system with no single correct answer.

Bounds. Output length within limits, latency within budget, token consumption within a ceiling, tool calls within a maximum count. Boring, and they catch runaway agent loops before your invoice does.

A property test is only as good as the property, and the discipline is to write properties that a plausibly wrong output would violate. "Response is non-empty" passes for confident nonsense.

Distributional testing

Some things are only true in aggregate. Run the same case many times and assert on the statistics rather than on any individual run.

This is the honest way to handle "the answer should usually be correct". Take a case, run it twenty or fifty times, score each run, and assert that the pass rate exceeds a threshold. Assert also on the spread: a case that passes ninety-five percent of the time is in a different operational category from one that passes ninety-five percent of the time with catastrophic output in the remainder.

Two cautions. Sampling costs money and wall-clock time, so reserve high-sample-count runs for a nightly job rather than every commit. And set thresholds from an observed baseline, not from a round number someone liked — measure the current configuration's pass rate first, then set the gate below it with enough margin to absorb sampling noise. A threshold tighter than your noise floor produces exactly the flakiness dynamic that destroys trust in a conventional suite.

Snapshot testing is a trap here

Snapshot testing — record the output, commit it, fail when it changes — looks made for this problem and is not.

It fails for a straightforward reason: the snapshot will change on every run for reasons unrelated to correctness, so the team learns to regenerate snapshots reflexively. Once "update the snapshots" is a habit, the snapshot suite has the same value as no suite, with additional repository noise. It is the flaky-test trust collapse arriving on schedule.

There is a narrow legitimate use. Snapshot the deterministic artefacts around the model: the rendered prompt, the assembled context, the retrieved document identifiers, the parsed structure. A diff in a rendered prompt is genuine information, because a prompt template change that nobody intended is a real and common defect. Snapshot the inputs to the model, never its outputs.

Where teams reach for snapshots of output, what they usually want is semantic similarity to a reference — which belongs in the evaluation harness with a scored rubric, not in the unit suite with an equality check.

What the pyramid becomes

The classic test pyramid is an argument about the cost of feedback, and that argument survives intact. What changes is that a new layer appears above the conventional ones, with different economics: it costs money per run, it is statistical rather than binary, and it is slow.

Layer	What it covers	Determinism	Where it runs
Unit	Prompt rendering, parsing, truncation, validation, tool logic	Fully deterministic	Every commit, seconds
Integration	Retrieval, model client with recorded responses, persistence, error paths	Deterministic via fixtures	Every commit, low minutes
Contract	Schema of model output, tool interfaces, provider response shapes	Deterministic	Every commit
Property	Invariants over live model calls, small sample counts	Probabilistic, asserted as properties	Pull request
Evaluation	Scored quality against a golden set and rubric	Statistical, compared to baseline	Pull request and nightly
Production	Live traces, outcome signals, drift monitoring	Reality	Continuously

The shape is still a pyramid, and the pressure is still to push each check to the cheapest layer that can honestly answer the question. What is new is that the top two layers cannot be made deterministic by trying harder, so they must be designed as measurements with baselines, thresholds and noise floors rather than as gates that are green or red.

What to do on Monday

Take your current suite and classify every test as deterministic or not. For most teams the deterministic pile is much larger than expected and much thinner than it should be. Fill the obvious gaps first: prompt rendering, context truncation at the budget boundary, response parsing against real malformed responses, and the retry path.

Next, write three property tests against a live model call. Start with schema validity, a grounding check, and a bound on tool calls or output length. Run each case ten times and assert the property holds every time. These three will catch more real defects than any equality assertion you could write.

Then go through the suite for equality assertions and output snapshots on model responses. Delete them or convert them: if the intent was structure, make it a property; if the intent was quality, move it to the evaluation harness where it can be scored against a baseline.

Finally, pin temperature and seeds in the test configuration and record the model identifier alongside every result. When behaviour changes next month, you want to be able to tell whether your code moved or the ground did.