Book a 30-min call
← cd ../blogs
$ cat posts/kv-cache-is-a-cache-configure-its-failures.mdx

KV cache is a cache: configure its failure modes

August 6, 2026 · ImmovableTech Team

  • Inference
  • Production AI
  • Real-Time Systems

A throughput switch that is actually a cache

Prefix caching gets adopted the way a compiler flag gets adopted. Someone reads that the serving engine can reuse the KV cache of a shared prompt prefix so a new request skips recomputing it, someone else notices that in TensorRT-LLM and SGLang it is on by default anyway, and the change lands with a note in the deploy channel about throughput. Nobody writes a runbook, because nobody thinks of it as a component.

It is a cache, with every property that implies, and you should be able to answer four questions about it in production: what is the hit rate, what does a miss cost, what is the eviction policy, and what happens to tail latency when the cache is cold. Most teams running prefix caching cannot answer any of the four, so their p99 is governed by a component they are not measuring and tuned by defaults they have not read.

We got this wrong on an agentic workload before we got it right. The failure modes below are the ones that cost us.

Two latency distributions wearing one number

A prefix cache hit skips prefill for the matched portion of the prompt. A miss pays for all of it. Those are not a good case and a slightly worse case of the same thing — they are two distributions, and your time-to-first-token metric is a mixture of them weighted by hit rate.

The mean is the one number that cannot see this. If 80% of requests hit and 20% miss, mean TTFT is dominated by the hits and looks excellent while p99 sits squarely inside the miss distribution, so tuning against the average optimises the path that was already fast.

Our own version of this was embarrassing. We built the dashboard from vLLM’s Prometheus counters, vllm:prefix_cache_hits over vllm:prefix_cache_queries, and reported the ratio as our hit rate. Both counters are expressed in tokens, not requests. A workload with a long shared preamble and a short variable tail reports a high token-weighted hit rate even when many individual requests get a partial match and still pay for a real prefill. Our token-weighted number was in the high seventies; the share of requests skipping prefill entirely was far lower, and that gap was exactly the population living in our p99. The counter is not wrong. We read it as something it does not measure.

Split the latency histogram by cache outcome and report the two separately. A single TTFT panel with a p99 line on it is not enough to operate this.

The cache key is your prompt layout

Reuse requires an exact shared prefix, so anything varying early in the prompt destroys reuse for everything after it. Teams poison their own hit rate without seeing the connection, because prompt layout is treated as a copywriting decision rather than a throughput one.

The usual culprits are a rendered timestamp, a session or request ID, a user’s display name, or a tool list serialised in whatever order the registry happened to return it. Any one of those near the top of a system preamble means every request has a unique prefix and the cache does nothing. Anthropic’s documentation spells the trap out directly: if your prompt is a large static context followed by a block containing a timestamp and the user message, and you set the cache breakpoint on that final block, you get no reuse, because writes happen only at breakpoints and the hash covering the breakpoint changes every request. The fix is to move the breakpoint to the last block that stays the same.

On OpenAI’s API the sensitivity extends past matching into routing. Requests are routed to a machine based on a hash of the initial prefix, typically the first 256 tokens, so a variable field inside that window does not merely miss — it sends you to a server that never had your prefix. That is what the prompt_cache_key parameter is for: it combines with the prefix hash to make routing stickier for traffic sharing a long common prefix.

The layout rule falls out of this and it is boring:

[ tool definitions          ]  stable across all requests
[ system instructions       ]  stable across all requests
[ retrieved context / docs  ]  stable across a session
--- cache breakpoint here ---
[ conversation history      ]  append-only
[ timestamp, IDs, user turn ]  varies every request

Sort the prompt by rate of change, most stable first, and put the breakpoint at the boundary. Two consequences get missed. Serialise tool definitions in a deterministic order, because a set iterated in hash order is a variable field wearing a stable field’s clothes. And be careful with sliding-window context truncation: the LMCache team’s production data found that trimming long inputs to the most recent tokens significantly reduces prefix cache hit ratios, because a truncated input no longer matches the prefix of anything cached. A context-management strategy chosen for correctness quietly became a throughput regression.

Eviction arrives exactly when the cache matters most

Cache capacity and batch capacity are the same memory. Blocks retained for future reuse and blocks held by in-flight sequences come out of one pool, and the serving stacks describe eviction as triggered by that competition rather than by age. SGLang’s radix tree evicts leaf nodes when memory is full, the policy exposed as --radix-eviction-policy and defaulting to LRU. TensorRT-LLM’s documentation puts the contention plainly: reusable blocks are displaced when the memory is needed for higher-priority work, such as propagating a request that is already running.

Eviction is triggered by memory pressure and memory pressure is caused by load, so the cache empties precisely during the traffic that most needs it. You can reach a state where adding concurrency reduces throughput: more in-flight sequences claim more of the pool, reusable blocks get evicted, the next requests miss and pay full prefill, prefill competes with decode for the same GPU, and the scheduler starts preempting. vLLM exposes vllm:num_preemptions for exactly this, and it is the most diagnostic counter on the page, because a rising preemption count under load says the system has entered a regime where your capacity model no longer holds.

The uncomfortable part is that this makes hit rate load-dependent, so a hit rate measured in a quiet hour tells you almost nothing about the hit rate during the incident.

Sharing a cache across users is a tenancy decision

Cross-request reuse is where most of the throughput comes from, and it deserves a deliberate decision, because what gets shared is derived from prompt content. Whether two requests may share a cache entry is a tenancy question that happens to be implemented in the serving layer, and the vendors treat it that way. Anthropic isolates caches between organisations and, on the Claude API and some platforms, between workspaces within an organisation, while noting that Bedrock and Google Cloud isolate at organisation level only — so the same code gets a different sharing boundary depending on where it runs. If you self-host, the boundary is whatever you built. vLLM’s documentation carries the relevant warning: the default prefix-caching hash is SHA-256, and switching to the faster non-cryptographic xxHash raises collision risk that could cause undefined behaviour or leak private information in multi-tenant environments. That is not a performance note, it is a threat model, and it belongs in a design review rather than a tuning session.

A shared prefix cache is fine when the shared content is genuinely shared — tool schemas, system instructions, public reference documents. The moment tenant-specific retrieved context enters the cached prefix you need per-tenant partitioning, or a written decision that those tenants sit inside one trust boundary.

Offloading buys a cheaper miss, not a free one

If your working set exceeds GPU memory, offloading moves blocks somewhere larger and slower instead of discarding them. TensorRT-LLM takes a hostCacheSize in bytes and copies reusable blocks into a host-memory buffer rather than evicting them, which greatly extends how long a block stays reusable. SGLang exposes a host tier through --hicache-size and pluggable storage backends behind --hicache-storage-backend.

The trade is real and not universally worth taking. NVIDIA’s documentation is unusually candid about where the line sits: the copy cost is negligible on Grace-Hopper, small enough to yield a net benefit for many x86 machines with Hopper GPUs, and unlikely to help on older architectures because the link between GPU and host memory is too slow. The same page notes the host buffer is pinned memory and that allocating a lot of it on x86 can take tens of seconds — a one-time cost, but one that lands at startup, which is also when your cache is cold.

Remote storage is what changes the calculus. The LMCache paper reports users loading KV cache from their own remote object store and achieving 22-32% lower TTFT than a full prefill, inverting the assumption that a remote fetch must be slower than recomputing. It also reports up to 15x throughput with vLLM on workloads like multi-round question answering — a workload with near-ideal prefix reuse, worth holding in mind before extrapolating.

Disaggregated prefill makes the failure explicit in a way we like. In vLLM’s NixlConnector, when the decode instance cannot load KV blocks from the prefill instance, kv_load_failure_policy decides what happens: fail is the default and errors the request, while recompute quietly runs the prefill on a decode-optimised instance and, as the docs warn, increases tail latency for every other request on that decoder. That is this whole post in one config key. The failure mode has a default, and if you have not read it you have accepted it.

What to emit, and how to test the cold path

Four metrics, and they are cheap. Hit rate as a rate over an interval rather than a lifetime gauge, which for vLLM means rate(vllm:prefix_cache_hits[5m]) / rate(vllm:prefix_cache_queries[5m]), with the token-weighting caveat stated on the panel. The proportion of requests that skipped prefill entirely, which is the request-weighted number and the one that predicts p99. TTFT split into two histograms by cache outcome, never merged. And eviction or preemption rate, which is the early warning that the other three are about to move.

Then test the cold path on purpose. A cache is only ever cold for reasons you can schedule: a deploy, a scale-out, a pod reschedule, a model swap. Restart a replica under representative load and watch p99 recover, because that number is your real worst case and you would otherwise meet it during an incident. Include a cache-hostile mix too — unique prefixes, no reuse — and confirm the system degrades rather than collapses. If cold p99 is outside your SLO, you do not have a caching problem, you have a capacity problem the cache was concealing.

For managed APIs the equivalent discipline is reading the current numbers rather than the ones you remember, because they differ by vendor, differ by model within a vendor, and change. As of August 2026 Anthropic’s minimum cacheable prompt length ranges from 512 to 4,096 tokens depending on the model, with a 5-minute default lifetime refreshed at no cost on each use and a 1-hour option; a 5-minute cache write is priced at 1.25 times base input tokens and a 1-hour write at 2 times, while reads sit an order of magnitude below base input on the published table. A prompt under the minimum is processed without caching and no error is returned. OpenAI caches automatically from 1,024 tokens in 128-token increments — a strict minimum on its newest model families, but varying between 1,024 and 2,048 on older ones, where prompts just above 1,024 tokens may not cache consistently. Same feature name, materially different contracts. Quote the doc and date the quote.

What we’d do differently

We would have treated the cold path as a first-class test before shipping rather than after. Our incident was self-inflicted and obvious in hindsight: a routine rolling deploy of an agent service, an empty cache on every new replica, and p99 TTFT at roughly four times steady state for several minutes while the shared tool-definition prefix was rebuilt. Nothing was broken. The system was simply doing at full traffic the work the cache had been hiding for weeks, and we had never measured that state because our load tests warmed up first, like everyone’s load tests do.

We would also have audited prompt layout when we enabled caching rather than months later. Our system preamble carried a rendered date near the top, put there so the model would not reason about stale timestamps — a reasonable decision made by someone with no idea it was also a cache key. It cost us most of the available reuse on one traffic class, and we found it by diffing two prompts rather than through any alert.

The general lesson is that we adopted a cache without adopting the habits that come with one. We would not ship a Redis layer without a hit-rate panel, an eviction policy we had chosen and a plan for a cold start. Prefix caching earned none of that scrutiny because it arrived as a flag. The discipline we apply to agent observability belongs here for the same reason: the parts of a system nobody instruments are the parts that decide your tail.

References

We tune inference serving and cache behaviour as part of our AI & Machine Learning Engineering practice. Talk to us if your p99 latency moves for reasons your dashboards cannot explain.