The Demo To Production Gap

Someone builds a prototype in two days. It answers questions about the company's documentation, or drafts replies to customer emails, or summarises a report. It is shown in a meeting. It works. The room is genuinely impressed, and they are right to be — the capability is real and a decade ago it would not have been possible at all.

Then somebody asks how long to put it in front of customers, and the engineer who built it says something like four to six months. The room does not believe this. From where they are sitting, the hard part is visibly finished. What remains must be paperwork.

This is the single most damaging misunderstanding in the field, and it damages things in both directions. Executives commit to timelines that cannot be met. Engineers, having been disbelieved, stop explaining and start absorbing the gap as personal failure. Projects get labelled as delayed when they were never scoped. The pattern is predictable enough that it is worth understanding structurally rather than treating each instance as a communication problem.

The gap exists for a specific reason. A demo is a system operating under conditions chosen by the person demonstrating it: benign inputs, one user, no time pressure, no cost ceiling, no consequence for a wrong answer, and a human present to interpret the output charitably. Production removes every one of those conditions simultaneously. The prototype was not a smaller version of the production system. It was a different system that happens to look similar.

What the demo did not have to handle

Go through the conditions one at a time. Each removal is real engineering work.

Inputs you did not choose. The demo used questions the builder knew the system could answer. Real users ask things outside the corpus, things that are ambiguous, things phrased in ways the documents never use, things in other languages, and things with typos. A meaningful share of real traffic is unlike anything in the demo, and much of it needs a graceful answer rather than a confident wrong one.

Users acting in bad faith. Somebody will try to make it say something embarrassing, extract its instructions, or use it for something unrelated to your product. If it reads untrusted content, somebody will try prompt injection. None of this was present in the demo, and all of it is present in week one.

Latency that users notice. The demo audience waited patiently for twenty seconds because they were watching something new. A customer will not. Response time becomes a budget you design against, which constrains how much retrieval you do, how many model calls you chain, and whether you can afford a verification step.

Cost that scales with success. The demo cost pennies. A feature used by every customer several times a day has a unit cost that must be smaller than the value it creates, and the architecture that makes that true is not the architecture of a prototype.

Consequences for being wrong. A wrong answer in a demo is an amusing moment. A wrong answer to a customer about their refund entitlement, their medication, or their contract is a complaint, a cost, and possibly a regulatory matter. The fraction of wrong answers that was tolerable at a demo is not tolerable in production, and closing that gap is most of the remaining work.

Everything failing. The provider will have an outage. A request will time out. A response will come back malformed. The index will be mid-rebuild. The demo had none of these because it ran once, successfully, on a laptop.

The work that remains

Here is the same argument as a list of things that must exist before launch. Almost none of it is visible in a demo, which is exactly why it is discounted.

Concern	Demo state	Production requirement
Quality measurement	Looked good to the builder	Golden set, rubric, scored regression suite
Latency	Whatever it took	Explicit budget, measured at the percentile you promise
Cost	Negligible	Unit cost per outcome, budgets, caps enforced in code
Observability	Console output	Traced requests with prompt, context, output, outcome
Failure handling	None	Timeouts, retries, fallbacks, degraded modes
Abuse	None	Rate limits, input filtering, output checks, monitoring
Versioning	The prompt in a file	Versioned prompts and models, deployable and reversible
Rollback	Undo in the editor	One action, no redeploy, tested
Access control	The builder's credentials	Per-user authorisation enforced at the data layer
Support	The builder	A team that can diagnose a complaint about an answer

Two of these deserve expansion because they are the ones most often left until last and most painful to add late.

Observability and tracing. When a customer complains about an answer, you need to reconstruct what happened: what they asked, what was retrieved, what prompt was actually sent, which model version responded, what it returned, and what the user did next. Without that record, every complaint is unfalsifiable and every fix is a guess. This is not a dashboard of request counts. It is a per-request trace you can open and read, and it is the difference between a team that improves its system and one that argues about it.

Prompt and model versioning. Prompts are logic. They change behaviour, they need review, and they need to be deployable and reversible independently of the application, because most of your production fixes will be prompt changes and waiting for a full release to ship one is intolerable. Pin the model version explicitly rather than accepting whatever an alias resolves to today, and record which version produced every response. A provider updating a model underneath you is a behaviour change you did not make and must be able to detect.

Degradation, not failure

A conventional service that cannot reach a dependency returns an error. Model-backed features have a wider set of options, and designing them is part of the work.

Fall back to a smaller or alternative model. Slower or less capable is usually better than nothing, provided the user is told.

Fall back to the non-model path. Search results instead of a synthesised answer. A template instead of generated text. Routing to a human instead of an automated reply. Very often the pre-existing feature is a perfectly good fallback and nobody thought to keep it.

Fail visibly and honestly. If the system cannot answer well, say so. The worst behaviour available is a confident, fluent, wrong answer produced because the retrieval step returned nothing and nobody wrote the refusal path. Design that path explicitly.

Cache to survive a short outage. Recent answers to common questions cover a surprising proportion of traffic during a provider incident.

Decide these in design, not during the incident. The question to answer for each feature is what the user should see when the model is unavailable, slow, or producing output that fails validation — three different situations that usually deserve three different behaviours.

Setting expectations without killing the idea

The conversation after a successful demo is a delicate one, because two things are true at once: the capability is real and worth investing in, and the timeline in everyone's head is wrong by a large factor. Lead with the first or you will be heard as the engineer who does not want to build the thing.

Some framings that work better than a list of objections.

Name the demo for what it is. A feasibility test, and a successful one. It answered the question "can this be done at all", which was worth answering. It did not answer "can this be done reliably, affordably, safely and supportably", which is a different question with a different budget.

Use the accuracy conversation to make it concrete. Ask what error rate is acceptable for this use case. Then show what the current system's error rate actually is on a realistic sample. That single comparison does more to convey the remaining work than any amount of architecture discussion, and it reframes the project as closing a measurable gap rather than as engineers asking for time.

Quantify the volume of failure. If the feature handles a thousand requests a day and is wrong five percent of the time, that is fifty wrong answers daily, every day. Ask who handles those and what each one costs. Executives who have only seen a demo have generally never done this multiplication.

Offer a staged path rather than a delay. Internal users first, then a small cohort, then general availability, with quality and cost measured at each stage and explicit criteria to proceed. This gives them something shipping soon, gives you the production data you need, and converts an argument about a date into a sequence of decisions with evidence attached. It is the same logic as progressive delivery applied to a system whose quality can only be measured on real traffic.

Be specific about what the remaining time buys. Not "hardening" or "productionisation", which sound like padding. Evaluation harness, tracing, fallback paths, cost controls, abuse handling, rollback. Each with a sentence about the failure it prevents.

What to do on Monday

Run the fifty-input exercise above on whatever prototype your organisation is currently excited about. Count how many outputs you would be comfortable sending to a customer unedited. That ratio is your starting quality, it is the number executives most need to see, and almost nobody has measured it.

Next, write down the answer to three questions for the feature as proposed: what is the acceptable error rate, what is the maximum acceptable response time, and what is the maximum acceptable cost per request. If nobody can answer these, the feature is not specified yet, and that is a finding worth reporting before any more engineering happens.

Then pick the two items from the production table that are most likely to be discovered during an incident rather than before one — usually tracing and a tested rollback — and do those before adding any further capability. They are the ones that determine how fast you can recover from everything else you have not thought of.

Finally, propose the staged rollout explicitly, with the criteria for each stage written down and agreed. It is the single most effective device for holding a conversation about a genuinely uncertain system with people who need dates, because it replaces one unreliable estimate with a sequence of checkpoints that everyone can see.