Book a 30-min call
← cd ../blogs
$ cat posts/speculative-decoding-at-production-concurrency.mdx

Speculative decoding at production concurrency: a latency trick, not a throughput one

July 16, 2026 · ImmovableTech Team

  • Inference
  • Production AI

The speedup is a property of your load, not of the technique

Speculative decoding is sold as free latency. A small draft model proposes several tokens, the big model verifies them all in one forward pass, and a rejection-sampling correction guarantees the output distribution is identical to what the target model would have produced alone. The benchmark numbers are real and the correctness argument is sound.

The problem is that nearly every headline number is measured at batch size one, and speculative decoding does not create speed out of nothing. It buys latency with compute, by running the target model over draft tokens that sometimes get thrown away. At batch size one your GPU is memory-bandwidth bound — it spends most of each decode step waiting on weights coming out of HBM — and that spare compute costs you nothing. Under continuous batching at production concurrency, the same GPU is already compute-saturated, and the wasted verification work now competes with your own throughput.

None of this is a discovery. Leviathan et al. wrote it into the limitations of the original ICML 2023 paper: latency improves “at the cost of an increased number of arithmetic operations”, and the method “is not helpful for configurations where additional computation resources are not available”. That sentence got lost somewhere between the paper and the vendor benchmark table. The engineering question is not whether speculative decoding works. It is where your traffic sits on the curve.

What acceptance rate actually is, and why it is domain-dependent

Acceptance rate is the single number that decides whether any of this pays. The draft proposes k tokens; the target verifies them in one pass and accepts the longest prefix consistent with its own sampling. Disagree at position three and you keep two tokens. The useful derived quantity is acceptance length — mean tokens emitted per target forward pass — because that is what divides your decode step count.

The original paper’s α values are a good calibration. Approximation models a couple of orders of magnitude smaller than the target landed between 0.5 and 0.9, and even a trivial bigram proposer reached α = 0.2 on English-to-German translation, which was still worth something because a bigram costs essentially nothing to run. The same paper reports α rising at temperature 0 relative to temperature 1: sharper output distributions are easier to guess.

That temperature effect generalises into what really determines your acceptance rate, which is entropy. Predictable text — boilerplate, structured output, code that follows an obvious pattern, a SQL statement whose shape is fixed by a schema — accepts well. High-entropy generation does not. This is why acceptance is a property of your workload and not something you can read off a leaderboard. On our restaurant intelligence platform, the NL-to-SQL path emits tightly constrained structured output, which is close to the best case; the free-form summarisation that sits next to it in the same service is not.

Acceptance also varies enormously within a workload, which the aggregate hides. The MLSys 2026 study of speculative decoding on vLLM measured per-request acceptance length on InstructCoder with Llama-3-70B and found EAGLE spanning 2.7 to 7.4 tokens, n-gram spanning 1.1 to 15.0, and a draft-model approach spanning 5.6 to 18.3. It decays within a request too: on GPQA-Main with GLM-4.5-Air, the multi-token-prediction head accepted at 0.92 for the first drafted token, 0.68 for the second and 0.38 for the third, because the released weights ship one MTP module reused autoregressively. Your third draft token is worth far less than your first, and a fixed draft length ignores that.

This variance is what makes speculative decoding awkward for continuous batching specifically. Every request in the batch emits a different number of tokens per step, so step time is set by the verification work you did, not the tokens you kept.

The method families, and which ones your serving stack has

Independent draft models are the original formulation: a separate small model of the same tokenizer family. They still win on large targets — the MLSys study found them strongest on the 70B target and weaker on 8B, because what matters is the cost ratio between draft and target, and a small draft is only cheap relative to a big verifier. You pay for it with a second set of weights to deploy and version.

Self-speculation and early exit remove the second model entirely. Draft & Verify (ACL 2024) drafts by skipping intermediate layers of the target and verifies with the full stack, requires no training and no extra memory, and reported up to 1.99× on Llama-2 variants in single-request benchmarks. It is elegant and it is not widely implemented in mainstream serving stacks, which is the practical reason we have never shipped it.

Medusa adds extra decoding heads to the target that predict several future positions in parallel, then verifies candidates with tree attention. It was the method that made draft-model-free speculation mainstream, and it has largely been overtaken. TensorRT-LLM’s docs still list Medusa but restrict it to Vicuna checkpoints; vLLM still exposes a medusa method. Neither is where new work is going.

The EAGLE line is where it went. EAGLE-3 (NeurIPS 2025) dropped feature-level prediction for direct token prediction with multi-layer feature fusion, which let draft quality scale with draft training data in a way EAGLE-1 and EAGLE-2 could not. Its abstract is also an unusually honest illustration of this post’s argument: it reports up to 6.5× speedup on academic benchmarks, and, in the same breath, 1.38× throughput in SGLang at batch size 64. Same method, same paper, a 4.7× difference in headline depending on which number you quote.

N-gram, or prompt-lookup, decoding proposes continuations by matching recent context against the prompt. It needs no training, no extra weights and no GPU memory at all, since the lookup runs over CPU-resident generation history. It is unremarkable on most workloads and excellent on ones with heavy input copying. The MLSys authors quantified exactly when: on code editing, once prompt-output BLEU-4 overlap exceeds roughly 0.6, n-gram beats EAGLE and EAGLE-3 at every batch size they tested, by up to 53% with a proposal length of 3 and up to 100% with a proposal length of 5. Below that threshold it underperforms, because it proposes confidently wrong tokens.

As of the current docs, vLLM supports n-gram, suffix decoding, draft models, EAGLE/EAGLE-3, MTP, MLP speculators and Medusa; SGLang supports EAGLE-2, EAGLE-3 and an n-gram mode; TensorRT-LLM’s PyTorch backend supports EAGLE-3 (with optional dynamic tree drafting), MTP, draft-target and n-gram, plus lookahead decoding for any autoregressive model. Availability is not the constraint any more. Choosing correctly is, and vLLM’s documentation now says so directly — it splits its recommendation table into “low QPS (latency focused)” and “high QPS (throughput focused)” columns, and describes n-gram and suffix decoding as giving “modest speedups without increasing workload during peak traffic”. The serving projects have internalised the load-profile argument. The posts citing them mostly have not.

Where the curve crosses

The MLSys 2026 study is, on its own account, the first systematic measurement of this on a production engine rather than a research prototype — five variants across four models and six workloads on vLLM, all on H100s, batch sizes 1 to 128. Its central result is monotone: absolute throughput rises with batch size, relative speedup falls.

For Llama-3.1-8B on GSM8K, EAGLE’s speedup goes from 1.73× at batch size 1 to 1.21× at batch size 128 on a single H100. The degradation is worse for bigger models, because they are already compute-bound at modest batch: on ShareGPT, going from batch 1 to batch 32 costs Llama-3.1-8B 4.3% of its speedup (1.68× to 1.61×) but costs Llama-3-70B on four H100s 14.0% (1.96× to 1.72×).

The number that should change how you configure things is about tree drafting. Tree-based EAGLE beats chain-based drafting at batch size 1 — on Llama-3-70B with ShareGPT, 1.81× for the chain against 2.03× for a 21-node tree. By batch size 64, the k=21 tree falls below 1× on every workload the authors tested, for both models, while the chain stays above 1× throughout. That is not a shrinking speedup. That is a configuration that makes your serving slower than turning speculation off, and it is the configuration that wins every batch-size-one benchmark.

The cleanest demonstration we have seen came from AMD’s ROCm team in June 2026, benchmarking EAGLE3 on a quantised Kimi-K2.5 across 8× MI325X with SGLang, on a 10,240-in/512-out workload. At concurrency 2, decode-step latency fell 68% (16.94 ms to 5.40 ms) and output throughput rose 175%. At concurrency 40, the same stack gave 36% lower decode latency and 33% more throughput. What makes this the useful data point is that acceptance was effectively unchanged between the two runs: accept length 3.97 at concurrency 2 and 3.93 at concurrency 40, against a ceiling of 4. The draft model was doing an identical job. The system had simply run out of spare compute to give it.

AMD and the vLLM project published a follow-up on 13 July 2026 covering EAGLE3 on MI355X, and its concurrency sweep shows the same slope more gently: on a 1K/1K workload with TP=4, an FP8 draft delivered 2.00× at concurrency 4, 1.87× at 16 and 1.76× at 64; a BF16 MiniMax-M2.5 draft fell from 1.79× at concurrency 4 to 1.38× at 64. The same post also isolates a second axis we had underestimated. Moving from 1K to 8K input, acceptance stays roughly stable but target verification cost per accepted token rises from about 0.30 ms to about 2.56 ms, and verification goes from 14% to 59% of draft-plus-verify time. Long context degrades speculative decoding for the same structural reason concurrency does, and the two compound.

The week we spent debugging the wrong thing

We enabled EAGLE-style speculation on a chat-shaped workload after benchmarking it the way everyone benchmarks it: one request at a time, watching time-per-output-token fall by more than half. In staging under real concurrency the p50 barely moved and aggregate throughput went down, and we spent most of a week assuming the draft model was the problem. We swapped draft checkpoints, tuned the draft length and considered training our own head.

Acceptance length had never changed. It sat around 3.5 tokens in every configuration we tried, at every concurrency. We were debugging model quality when the problem was that our GPUs had no idle compute left to donate — the shape the AMD MI325X figures above describe exactly.

The second thing we got wrong was assuming summarisation meant n-gram would win, on the reasoning that summaries copy from the source. Ours were abstractive — the model was rewriting, not extracting — and prompt-output overlap sat well below the threshold where lookup helps. n-gram proposed plausible-looking spans that the target rejected, and it cost us a few percent. “Heavy input copying” is a measurable property, not a genre.

How we decide now

We no longer treat this as a model choice. The first question is what concurrency the service actually runs at, at the percentile that matters, and the answer is usually already in the serving metrics.

If a service runs at low concurrency and latency is what users feel — an interactive coding assistant, a voice turn, a single-user agent loop where each step blocks the next — speculation is one of the highest-return things available and a model-based method like EAGLE-3 or native MTP is worth the deployment cost. If a service runs saturated and throughput is the constraint, the honest default is either no speculation or n-gram, which is close to free and can be enabled per workload. In between, you measure.

Measuring is cheap, and it is the step nobody does:

for c in 1 4 16 64 128; do
  vllm bench serve --backend vllm --model "$TARGET" \
    --dataset-name random --random-input-len 1024 --random-output-len 512 \
    --num-prompts $((c * 10)) --max-concurrency "$c" --request-rate inf \
    --percentile-metrics ttft,tpot,itl,e2el
done

Run that sweep with speculation off, then again with it on, and read the crossover off the two curves. Use your real traffic shape rather than a random dataset if you can; input length and output entropy both move the answer. Then set the config to whatever the sweep says:

from vllm import LLM

llm = LLM(
    model="target-model-path",
    speculative_config={
        "method": "eagle3",
        "model": "draft-model-path",
        "num_speculative_tokens": 2,
    },
)

Keep num_speculative_tokens small unless the sweep justifies more. Given how sharply position-wise acceptance decays, the marginal draft token is usually verification cost you pay and throw away.

What we’d do differently

We would benchmark at production concurrency first and at batch size one never, or at least not as anything other than an upper bound. Every hour we spent on draft-model quality before we had a concurrency sweep was wasted, because the sweep would have told us in twenty minutes that the technique was fighting our load profile rather than that our draft was bad.

We would also instrument acceptance length as a served metric rather than a benchmark artefact, and per route rather than globally. It is the one number that tells you whether a regression came from a model change, a prompt-template change or a traffic-mix change. Our summarisation and structured-output paths shared a deployment and a single aggregate acceptance figure, which described neither of them.

And we would stop reading unqualified speedup claims. A speculative decoding figure without a batch size and a hardware configuration attached is not a weak claim, it is not a claim at all — the same method in the same paper can be 6.5× or 1.38× depending on which line you quote.

References

We tune inference stacks against real load profiles as part of our AI & Machine Learning Engineering practice. Talk to us if your serving benchmarks and your production latency disagree.