Inference Cost As An Architectural Constraint

Conventional software has a cost curve that engineers have internalised so thoroughly they no longer think about it. Building the feature is expensive; running it is close to free. Marginal cost per request rounds to nothing, capacity is bought in lumps, and a feature that becomes ten times more popular is a success story with a hardware conversation attached.

Model-backed features invert this. The marginal cost of a request is real, it is paid every single time, and it scales linearly with usage. A feature that succeeds beyond expectations produces a bill that also succeeds beyond expectations. This is not a procurement problem to be handled later by negotiating a better rate. It is an architectural constraint, and like every architectural constraint it is cheap to accommodate early and expensive to retrofit.

The retrofit problem is the important part. Most of the decisions that determine unit cost — how much context you assemble, whether prompts are structured to be cacheable, whether a request can be routed to a smaller model, whether your evaluation harness can tell you if a cheaper configuration is acceptable — are made in the first weeks and become load-bearing. By the time finance asks the question, the system has been built on the assumption that every request goes to the largest available model with everything the team could think of stuffed into the context, and unwinding that is a rewrite.

Treat cost the way you treat latency: a budget set during design, measured continuously, and enforced.

Know what you are actually paying for

Before optimising anything, be precise about where the money goes, because the intuitions are wrong in a specific way.

Cost is driven by tokens, split between input and output, and these are not priced alike — generated tokens cost substantially more than tokens you supply. That asymmetry has a direct design consequence: a system that reads a great deal and writes a little is cheap relative to one that generates at length, and verbose output formats are a recurring and invisible cost.

Input volume is usually where the waste is, because input grows silently. A conversation accumulates history. A retrieval step that returns ten chunks instead of five doubles a large part of the prompt. Few-shot examples added during tuning are never removed. System instructions grow as edge cases are patched. None of these changes feels like a cost decision at the moment it is made.

Then there are the multipliers. An agent loop sends the accumulated context again on every iteration, so a ten-step run with a growing context costs far more than ten times a single call. Retries multiply. Evaluation runs multiply — a nightly harness over a large golden set with multiple samples per case is a real line item. Self-critique, reranking with a model, and guardrail checks each add a call that nobody counted when the feature was specified.

The measurement that matters is not cost per call. It is cost per completed outcome: per resolved support conversation, per document processed end to end, per successful agent run including the ones that failed and were retried. That number is comparable to the value of the outcome, and it is the only version of the number that tells you whether the feature is viable.

Context budgeting

Long context windows are presented as a convenience and used as an excuse not to think. The temptation is to include everything that might be relevant and let the model sort it out. This costs money on every request, adds latency, and frequently makes output worse, because relevant material gets diluted by irrelevant material and models attend unevenly to very long inputs.

Set an explicit token budget per request, allocate it deliberately, and enforce it in code.

Trim conversation history rather than accumulating it. A rolling window of recent turns plus a periodically refreshed summary of what came before costs a fraction of the full transcript and is usually indistinguishable in quality. The full history remains in your own store; it does not have to be re-sent every turn.

Retrieve fewer, better chunks. This is the argument for reranking made on economic grounds. Retrieving fifty candidates cheaply and passing five good ones to the generator costs less at the expensive step than passing twenty mediocre ones, and it usually scores better. Cost and quality point the same direction here, which is rare enough to be worth exploiting.

Compress what you pass. Strip boilerplate, navigation, repeated headers and markup from retrieved documents before they enter the context. Summarise long tool results rather than pasting them whole. A verbose API response consumed raw can dominate a prompt.

Prune instructions on a schedule. System prompts accumulate rules added to fix specific incidents, most of which are never removed and some of which no longer apply. Review them against your evaluation harness: remove a section, run the set, see whether the score moves. This is one of the few genuinely free optimisations available.

Constrain output length. Ask for the structured fields you need rather than prose you will parse. Since generated tokens are the expensive ones, terse output formats pay twice — in cost and in latency.

Caching is the largest single lever

Caching in this context means three quite different things, and teams usually implement the least valuable one first.

Exact-match response caching. Identical input returns a stored response. Trivial to build, and the hit rate is low for conversational traffic and high for anything with repeated queries — document classification, enrichment pipelines, common support questions. Worth having; rarely transformational alone.

Semantic caching. Embed the query, look for a previous query close enough in embedding space, return that response. The hit rate is far better than exact match. The risk is precise: two questions can be semantically similar and require different answers, and a threshold loose enough to be useful will eventually serve a wrong answer confidently. Use it where answers are stable and general, never where the answer depends on user-specific state, and evaluate the threshold against a labelled set rather than picking a number that felt right.

Prompt prefix caching. The provider retains the processed form of a repeated prompt prefix and charges less for it on subsequent requests. This is the one to design around, because it changes how you order a prompt. Put everything stable at the front — system instructions, tool definitions, few-shot examples, any long document you will ask multiple questions about — and everything variable at the end. A single dynamic value near the top of an otherwise static prompt invalidates the whole prefix.

That ordering requirement is exactly the kind of thing that is easy on day one and awkward on day two hundred, because by then the prompt is assembled by four different code paths that each insert something. Structure the assembly so the stable-then-variable ordering is enforced by construction.

Routing and cascades

The default architecture sends every request to one model, chosen for the hardest case the team could imagine. Most traffic is not the hardest case, and paying the hardest-case price for the median request is the most common structural inefficiency in these systems.

Routing classifies the request up front and sends it to an appropriately sized model. Simple extraction, classification and formatting are handled well by smaller models. Complex multi-step reasoning goes to a frontier model. The classifier itself must be cheap — a small model, or often a deterministic rule on request type, which works better than people expect.

Cascading tries the cheap model first and escalates when the result is inadequate. This requires a reliable inadequacy signal, and the quality of that signal determines whether the pattern works. Deterministic checks are the good case: schema validation failed, a required field is missing, no citation was produced, a confidence field came back low. A model-based quality check as the escalation trigger is usually a false economy, because you have added a call to decide whether to add a call.

Cascades are worth it when the cheap model handles a large majority of traffic. When escalation is frequent you are paying for both models on most requests and have added latency for nothing. Measure the escalation rate before committing to the pattern.

Approach	Cost profile	Latency	Main risk
Single frontier model	Highest, flat	Predictable	Overpaying for easy traffic
Routing by classifier	Lower, depends on mix	Small added step	Misrouted hard cases silently degrade
Cascade with escalation	Lowest when escalation is rare	Worse for escalated requests	Poor escalation signal; both models paid
Fine-tuned small model	Lowest at volume, plus training cost	Best	Retraining burden, drift, narrower scope

Batching applies wherever work is not interactive. Overnight classification, backfills, bulk enrichment and large evaluation runs do not need a synchronous response, and asynchronous batch processing is priced accordingly. Building an offline path is straightforward at design time and awkward later, because by then everything is wired through a synchronous request handler.

When a smaller or fine-tuned model is the right answer

The frontier model is the correct starting point. It is the fastest route to knowing whether the task is solvable at all, and it removes model capability as a variable while you get everything else working. It is not automatically the correct steady state.

Consider a smaller or fine-tuned model when the task is narrow and well defined, when volume is high enough that unit cost dominates development cost, when latency matters to the user experience, when you have accumulated a decent dataset of good outputs from the larger model, and — the precondition that makes the rest possible — when you have an evaluation harness capable of telling you whether the smaller model is acceptable.

That precondition is not negotiable. Without a harness, substituting a cheaper model is a change you cannot evaluate, so it will either be blocked indefinitely by nervousness or shipped and discovered by customers. The harness is what makes cost optimisation a routine engineering decision rather than a gamble, and it is the reason cost work tends to be available to teams with good evaluation practice and unavailable to everyone else.

Be honest about the ongoing burden. A fine-tuned model is an asset you own and must maintain: retraining as the task evolves, re-evaluation as the base model changes, and a deployment path. Quantisation and distillation reduce serving cost further and trade some quality, which your harness can quantify. None of this is free, and for low-volume features none of it is worth doing.

What to do on Monday

Work out your cost per completed outcome for the largest model-backed feature you run. Not per call — per resolved conversation, per processed document, including retries and failures. Compare it to what that outcome is worth. Most teams have never produced this number, and producing it takes an afternoon if you have token instrumentation and a week if you do not, which is itself the finding.

Then take a sample of real production prompts and read them. Measure what fraction of the input tokens is conversation history, retrieved content, instructions and examples. Find the largest component and ask whether it is earning its place, testing the answer against your evaluation set rather than against opinion.

Reorder your prompt assembly so everything stable precedes everything variable, and verify against your provider's prefix caching behaviour that you are getting the discount. This is usually the highest return for the least effort available.

Finally, add a per-request and per-run cost ceiling that is enforced in code and alerts when it is approached. Agent loops and retry paths are the common sources of runaway spend, and a counter stops them. Then set a cost budget per feature alongside the latency budget, and review both in the same conversation, because they trade against each other and deciding one without the other is how you end up with a system that is fast, good and unaffordable.