Book a 30-min call
← cd ../blogs
$ cat posts/inference-savings-are-a-procurement-decision.mdx

Inference savings are a procurement decision, not a kernel problem

July 30, 2026 · ImmovableTech Team

  • Inference
  • Production AI

The optimisation that got refunded

On 30 July 2026 OpenAI cut the API price of its cheapest GPT-5.6 tier by 80% and its mid tier by 20%, leaving the top tier untouched. Its own changelog dates the release of that model family to 9 July 2026. Twenty-one days elapsed between shipping a model and repricing it to a fifth of its launch rate.

If your team had spent those same three weeks on serving-stack work and landed a 30% throughput improvement — a genuinely good quarter’s result — you would have taken 23% off that portion of the bill. The vendor took off 80%, roughly three and a half times as much, and you would have had to do nothing to collect it.

We are not arguing that inference optimisation is worthless. We are arguing about ordering. For most teams the dominant terms in the inference bill are which model tier you route to, which vendor and precision you buy, whether you committed to capacity, and above all how many tokens your product design requires per user action. Those are procurement and architecture decisions. Kernel-level work matters at the point where you have already made them well, and teams routinely invert that order because the engineering feels more like engineering.

The utilisation floor is what decides self-hosting

The self-hosting question is usually argued in terms of throughput. It is actually decided by utilisation, because a reserved accelerator costs the same whether or not you are using it.

The structure is three numbers. You have a fixed cost per hour for the reserved instance. You have an achievable output-token rate at your latency target, which you must measure yourself. And you have the API price per million tokens you are trying to beat. The break-even is the fraction of each hour that box must be serving at that measured rate before self-hosting wins.

def break_even_utilisation(
    instance_usd_per_hour: float,
    output_tokens_per_second: float,
    api_usd_per_million_tokens: float,
) -> float:
    """Fraction of each hour a reserved instance must serve at the measured
    rate before self-hosting beats the API. Above 1.0, the API always wins."""
    million_tokens_per_hour = output_tokens_per_second * 3600 / 1_000_000
    api_cost_of_same_output = million_tokens_per_hour * api_usd_per_million_tokens
    return instance_usd_per_hour / api_cost_of_same_output

Put AWS’s published rates through it. An EC2 Capacity Block for a p6-b200.48xlarge — eight NVIDIA B200s — is $98.84 per instance-hour in US East, or $12.355 per accelerator-hour, with Linux billed at zero on top. Against an output price of $1.20 per million tokens, breaking even at full utilisation requires roughly 22,900 output tokens per second sustained across those eight GPUs. At 40% utilisation you need about 57,000. Against a frontier-tier output price of $30 per million, the same instance breaks even at roughly 915 tokens per second.

Same hardware, same formula, same rate card. The answer moves by a factor of 25 purely because of which model tier you decided your product needed. That is the whole argument in two numbers, and no amount of kernel work closes a 25× gap.

Two honest caveats, both of which make self-hosting look worse than the arithmetic above. We compared against the output price alone; real traffic also bills input tokens at a much lower rate, so the blended API price is lower than the number we used. And the instance rate is not the cost — it excludes storage, egress, redundancy for the times a node fails, and the engineers who keep it serving.

There is also a structural reason the floor is hard to clear that has nothing to do with your competence. The API vendor pools demand across thousands of customers, so their accelerators run near saturation while yours follow your traffic curve and idle overnight. You are not competing with their engineering. You are competing with their utilisation, and you cannot buy that.

Self-hosting still wins in specific shapes: steady batch workloads with no diurnal curve, data-residency constraints that price the API out entirely, a fine-tuned small model with no API equivalent, or genuinely enormous sustained volume. Notice that every one of those is a statement about demand, not about serving code.

Low-precision economics are a hardware purchase

Precision is the other term people treat as an engineering choice and it is mostly a procurement one, because the arithmetic units either exist in the silicon you rented or they do not.

NVIDIA’s published DGX B200 specifications give the eight-GPU system 144 PFLOPS of FP4 Tensor Core throughput against 72 PFLOPS of FP8 — 18 and 9 PFLOPS per B200. Read the footnote before you plan around those figures: they are quoted with sparsity, and dense performance is half the number shown. What the table actually tells you is that each halving of precision doubles the paper ceiling, and only the paper ceiling. Real serving throughput is set by memory bandwidth, batch composition and how much of your model you left in higher precision, and it does not track the ratio.

The hardware gate is hard. The vLLM project’s LLM Compressor documentation lists FP8 weight-and-activation quantisation as requiring compute capability 8.9 — Ada Lovelace and newer — while both NVFP4 and MXFP4 require compute capability 10.0, meaning Blackwell. If you are on Hopper, four-bit floating point is not a tuning exercise you have been neglecting. It is unavailable, and the only route to it is a purchase order.

The two four-bit formats are not interchangeable and the difference is not speed. Per the same documentation, NVFP4 uses NVIDIA’s two-level micro-block scaling and requires calibration data to compute activation global scales, while MXFP4 uses per-group quantisation with a group size of 32 and E8M0 scales, needs no calibration data under round-to-nearest, and is cross-platform via the OCP microscaling specification. You are choosing between vendor-specific accuracy machinery and portability across accelerator vendors — a supply-chain decision.

On what precision costs in quality, we are going to decline to give you a number. The published deltas we found were either vendor-produced or single-repository claims, and quantisation error is workload-specific enough that someone else’s perplexity table tells you very little about your extraction pipeline. What we will pass on is the framework maintainers’ own framing, which is more useful than a benchmark: FP8 recovers accuracy with plain round-to-nearest and no calibration data, whereas four-bit formats may need GPTQ or AWQ and a calibration set to recover. That difference in required effort is the honest signal. FP8 is close to free; FP4 is a project with an evaluation budget attached.

The cheapest token is the one you never generate

The demand side is where the largest recoverable money sits, and it is the side with no interesting engineering in it, which is presumably why it gets skipped.

Start with caching, because the vendors have already told you what it is worth. Anthropic’s list prices effective 27 May 2026 put an Opus-tier cache hit at $0.50 per million tokens against a $5.00 base input rate. OpenAI’s rate card applies the same 90% discount to cached input reads. Both vendors are offering to bill your repeated prefix at a tenth of list, and collecting that requires no kernel work at all — only that your prompts put the stable material first and stop reordering system content between requests. We have watched a team lose most of a cache discount because a timestamp was being interpolated near the top of a prompt template.

Then the multipliers people forget are multipliers. On OpenAI’s rate card a long-context request bills at double the input rate and one and a half times output, so the same conversation crosses a threshold and silently doubles. Anthropic’s table prices US-only inference at $5.50 against $5.00 global for the same model, and prices batch processing at exactly half the standard rate. Context length, data residency and latency tolerance are not architectural details; they are line items, and each one is a decision someone on your team made without seeing the price.

The biggest lever is still output length and routing. Every token a frontier model generates costs several times what the same token costs from a small one, and most production traffic does not need the frontier tier for most of its steps. This is a familiar failure to us from agent work: we have written before about MCP systems in production where an agent made 40 tool calls per request instead of the expected 4 to 6, and the fix was a per-run token budget rather than anything in the serving path. A capped output length and a router that sends easy steps to a cheap model will beat a quarter of serving optimisation in almost every system we have measured.

Optimisation has a shelf life shorter than the work

This is the part that is uncomfortable to write down. Epoch AI’s price-trend analysis — which publishes its method, fits a log-linear regression to the cheapest model clearing each capability threshold, and explicitly excludes reasoning models because their token counts distort per-token comparisons — found prices at fixed capability falling between 9× and 900× per year across six benchmarks, with a median of 50×. Restricted to data after January 2024 the median rises to 200× per year. Even the slowest trend in that range outruns anything a serving team ships in a quarter.

The other way optimisation expires is that it was measured at the wrong operating point. The MLSys 2026 study of speculative decoding on vLLM is the cleanest published example: for Llama-3.1-8B on GSM8K, EAGLE’s speedup fell from 1.73× at batch size 1 to 1.21× at batch size 128, because at large batch the system is already compute-saturated and the compute spent proposing rejected tokens stops being free. On ShareGPT with a 70B target, moving from batch 1 to batch 32 alone eroded the speedup by 14%, from 1.96× to 1.72×. Read that as a rule about your own benchmarks: a speedup measured at low concurrency is a number that shrinks as your product succeeds.

We learned this the way you would expect. We spent the better part of a month tuning speculative decoding against a harness that ran effectively one request at a time, presented a good multiple to a client, and then watched most of it dissolve when the same configuration went behind real concurrent traffic. Nothing was wrong with the implementation. We had optimised the wrong operating point, and the published literature now says so more precisely than our post-mortem did.

So which work durably pays? The dividing line we now use is simple. Anything that reduces tokens or raises utilisation survives model releases, price cuts and hardware generations, because it changes the quantity you buy rather than the unit price. Prompt structure, cache-hit rate, output caps, model routing, request batching and consolidating traffic onto fewer, busier replicas all sit on that side. Anything that hand-tunes around one model’s quirks or one serving version’s behaviour is depreciating from the day you merge it, and the depreciation schedule is set by someone else’s release calendar.

What we’d do differently

We would compute the break-even utilisation before the first architecture meeting, not after a proof of concept had already been built on reserved hardware. On one engagement we sized a deployment against peak throughput, and because the traffic had an ordinary business-hours curve the box sat idle for most of the day. The effective cost per token was several times the headline figure we had put in the plan. The number that would have caught it took ten minutes to calculate and we calculated it in month three.

We would instrument cache-hit rate on day one. Prefix caching is presented as a free win and is in practice a cache, with a hit rate that quietly collapses when a prompt template changes. Untracked, it degrades invisibly, and the only symptom is the bill.

We would set the token budget before choosing the model, rather than choosing a model and then discovering what it costs to run the product we designed. Almost every expensive system we have inherited was expensive because of a product decision — full transcripts resent every turn, unbounded agent loops, one model tier for every step — that nobody priced at the time it was made.

And we would stop treating vendor price movements as noise in the plan. They are the largest single term in most forecasts we have written. Build the cost model so that a tier swap or a price change is a configuration edit, keep the routing decision on your side of the API, and you get to collect the next 80% cut without a migration.

References

We size and cost production inference systems as part of our AI & Machine Learning Engineering practice. Talk to us if your inference bill is growing faster than your traffic.