Evaluation Harnesses For LLM Systems

There is a meeting that happens in every organisation building on language models, usually about six weeks after the first prototype impressed someone. An engineer has changed the prompt. A product manager thinks the new output is worse. A second engineer thinks it is better. Someone pastes three examples into a channel. Someone else pastes three different examples. The discussion runs for forty minutes and ends with the change being merged because the person who made it was more senior, or reverted because the person who disliked it was.

That meeting is the symptom. The disease is that the team has no instrument capable of answering the question. In conventional software this situation does not arise, because behaviour is deterministic and a test either passes or it does not. In a system whose central component produces a distribution over plausible outputs, "better" is a statistical claim, and statistical claims cannot be settled by anecdote no matter how senior the anecdote is.

An evaluation harness is the instrument. It is a golden dataset, a scoring method, a runner and a record, and it converts prompt and model changes from matters of taste into matters of evidence. It is also, consistently, the part of the work teams defer. Building it is unglamorous, it produces no demo, and it feels like overhead right up until the week you need to change the model underneath a live product and discover you have no way to know whether you have broken anything.

Build it early. It is cheaper than the arguments.

The golden set is the asset

Everything else in the harness is replaceable. The dataset is not.

A golden set is a collection of inputs paired with what a good output looks like, curated by people who understand the domain. Its value is not in its size. A hundred well-chosen cases will teach you more than ten thousand scraped at random, because the ten thousand will be dominated by the easy middle of the distribution and you already know the system handles that.

Start from real traffic, not imagination. Synthetic test cases written by the team who built the system encode the team's assumptions about how it will be used. Users do not share those assumptions. If you have production logs, sample from them. If you do not, sample from whatever the closest real artefact is — support tickets, search queries, existing documents — before you invent anything.

Stratify deliberately. You want the common cases, because regressions there are the ones that damage trust at volume. You want the known-hard cases, because that is where improvement is available. And you want the adversarial cases: ambiguous inputs, inputs where the correct answer is "I do not know", inputs that attempt to make the system do something it should refuse. A set with no refusals in it will happily reward a model that never refuses.

Include the failures you have already seen. Every incident, every complaint, every bug report becomes a case. This is the same discipline as adding a regression test when you fix a defect, and it has the same effect: the set gets more valuable in exactly the places where you have been hurt.

Version it and hold some back. The set lives in source control alongside the code. Keep a portion sealed and evaluate against it rarely, because a set you iterate against continuously is a set you will eventually overfit to — you will tune prompts until they score well on those specific hundred items and learn nothing about the eleventh hundred.

Budget real time for this. Curating a few hundred good cases with domain experts is days of work, not hours, and the people who can do it are usually the people with the least free calendar. That is the actual cost of an evaluation harness, and it is worth paying.

Rubrics: decide what you are measuring before you measure it

A rubric turns a vague quality judgement into a set of specific, separately scored questions. The temptation is to score "quality" on a scale of one to five. Resist it, because a single number aggregates dimensions that move independently and hides the trade you actually made.

Split it. For a question-answering system a workable rubric might score factual accuracy against the provided sources, whether every claim is attributable, whether the question was actually answered, adherence to format, and tone. A change that improves fluency while degrading attribution is a bad change; a single composite score can easily report it as an improvement.

Prefer binary or three-point scales over five and seven-point ones. Human raters — and model raters — are unreliable at fine distinctions, and the difference between a three and a four is where your measurement noise lives. "Correct / partially correct / wrong" is more repeatable than a seven-point scale, and repeatability is the whole point.

Write the rubric as instructions precise enough that two competent people applying it independently reach the same verdict. If they do not, the rubric is not finished yet. That test is not rhetorical — it is the next section.

LLM-as-judge, and why you validate the judge

Human scoring does not scale to every commit. The standard answer is to use a model to apply the rubric, which works well enough to be indispensable and badly enough to be dangerous if you trust it without checking.

The failure modes are well documented and all of them will bite you.

Position and verbosity bias. Judges shown two candidate answers tend to favour whichever is presented first, and tend to favour the longer, more elaborate one regardless of whether the extra content is correct. Randomise presentation order and check whether your judge's scores correlate with output length.

Self-preference. A judge tends to rate output from the same model family more highly. If you are using the same model to generate and to judge, you have built a system that grades its own homework.

Sycophancy towards the reference. Give the judge a reference answer and it will often reward surface similarity to that reference rather than correctness, penalising a different but equally valid answer.

Insensitivity to the failures you care about. Judges are reasonably good at fluency and structure, weaker at subtle factual error, and poor at catching a confidently stated fabrication when the surrounding text is coherent. Which is, of course, exactly the failure you most want caught.

The discipline that makes this usable is straightforward: treat the judge as a measuring instrument and calibrate it against human labels. Take a sample of your golden set, have two or more domain experts score it independently using the rubric, and measure how well they agree with each other. That inter-rater agreement is your ceiling — the judge cannot be more reliable than the humans defining the target. Then measure agreement between the judge and the human consensus on the same items.

If the judge agrees with humans roughly as often as humans agree with each other, you can run it automatically and spot-check. If it does not, the fix is usually the rubric rather than the model: ambiguous criteria produce disagreement in humans and judges alike. Re-calibrate whenever you change the judge model, the judge prompt or the rubric, and keep the human-labelled calibration sample as a fixed artefact.

Put it in the pipeline

An evaluation you run when you remember to is not a gate. Wire it into continuous integration and treat it like any other test suite, with adjustments for the fact that it is statistical and it costs money per run.

Trigger	Scope	What it gates
Every commit touching prompts or model config	Small fast subset, deterministic settings	Blocks merge on regression
Pull request	Full golden set, judge-scored	Blocks merge on score drop beyond threshold
Nightly	Full set plus adversarial and safety cases	Raises an issue, notifies owner
Pre-release	Full set plus held-out set	Blocks the release

Two design decisions matter more than the schedule.

First, compare against a baseline rather than an absolute threshold. What you want to know is whether this change moved the score relative to the currently deployed configuration, on the same data, scored the same way. Absolute thresholds invite the team to tune the threshold.

Second, set the threshold wide enough to absorb noise. Run the baseline twice and see how much the score moves with no change at all; that is your noise floor, and a gate tighter than it will fail randomly and be disabled within a month — the same trust collapse that flaky tests produce in a conventional suite.

Version everything that affects the result: prompt text, model identifier, retrieval configuration, temperature, the rubric, and the judge prompt. A score without that metadata is not reproducible, and an irreproducible measurement is a rumour.

Offline scores are not production quality

The harness measures your system against a dataset you chose. Production measures it against reality. These diverge, and the gap is where most unpleasant surprises live.

Offline evaluation systematically misses the distribution shift between your curated inputs and what users actually send, the effect of real latency and timeouts on real sessions, interaction effects in multi-turn conversations that single-turn cases never reach, and the slow drift of the underlying content your system depends on.

So instrument the live path as well. Capture the inputs, the retrieved context, the prompt actually sent, the output and the outcome for every request — the same tracing discipline described in putting AI in the delivery pipeline. Collect implicit signals: did the user rephrase immediately, did they abandon, did they copy the answer, did they escalate to a human. Collect explicit feedback where you can get it, while remembering that thumbs-down is heavily biased towards visible failures and silent about quiet ones.

Then close the loop in both directions. Sample production traffic, have humans label it, and check whether your offline scores predict those labels — that correlation is the thing that tells you your harness is worth running. And feed the interesting production failures back into the golden set, so the set tracks reality rather than the reality you assumed eighteen months ago.

What to do on Monday

Collect fifty real inputs from whatever source you have — logs, tickets, a week of queries. Do not clean them up. Run your current system against all fifty and read every output yourself. This takes about two hours and will find problems you did not know you had.

Write the rubric next, with three to five separately scored criteria and a three-point scale. Have two people score the same fifty outputs independently, then compare. Where they disagree, the rubric is ambiguous; fix the wording and repeat until they mostly agree. Those fifty human-labelled items are now your calibration set.

Then automate the runner — an input, the system under test, a judge applying the rubric, a score written somewhere durable — and measure the judge against your fifty labels. Only once that agreement number is acceptable should you put the harness in the pipeline as a gate.

Finally, make it a standing rule that no prompt or model change ships without a score on the scoreboard. Not a policy document, just the rule. The first time it catches a change that everybody believed was an improvement, it will have paid for itself.