The single most counterintuitive thing about LLM inference is how badly we waste our hardware. I have spent the past several years helping teams stand up large-model serving infrastructure, and the pattern I keep seeing is teams who buy H100s expecting compute-bound workloads, then discover their GPUs are sitting at 30-40% utilization while the bottleneck is memory bandwidth. A hundred billion parameter model generates one token per forward pass. The GPU finishes the arithmetic in milliseconds, then waits while weights stream in from HBM. Every GPU cycle spent waiting is money evaporating.
Speculative decoding attacks this problem directly. It does not change the model, it does not reduce quality, and it does not require you to buy different hardware. What it does is restructure the inference loop so you generate multiple tokens per target-model forward pass, converting what was a serial memory-bandwidth bottleneck into something your compute units can actually chew on. Done right, and with the right workload profile, it delivers the kind of throughput improvement that normally requires a hardware upgrade.
This is what EAGLE-3 and P-EAGLE have achieved in production serving in 2026. Let me explain exactly how it works, when to use it, and how to configure it in vLLM and SGLang without wasting two weeks on mistuned parameters.
The Core Mechanism: Draft Once, Verify in Parallel
The fundamental insight behind speculative decoding is that verification is cheaper than generation. Generating token N+1 requires a full autoregressive forward pass through the target model; verifying whether a proposed token N+1 is acceptable given a known prefix can piggyback on a batched forward pass that checks multiple candidates simultaneously.
Here is how the loop works. A lightweight draft model proposes K candidate tokens sequentially: it takes your current context and generates token 1, then token 2 conditioned on token 1, and so on up to K tokens. This draft model is much smaller than the target model, so it generates candidates quickly. The target model then runs a single forward pass over the entire context plus all K candidates simultaneously and either accepts or rejects each candidate using rejection sampling. Critically: if the target model rejects candidate token i, it generates the correct token at position i and discards all subsequent candidates. If it accepts all K candidates, you get K+1 tokens for the cost of approximately one target-model forward pass (plus the cheap draft passes).
The expected speedup is the product of the mean accepted length (how many candidates the target model accepts on average before rejecting) and the ratio of target-to-draft model cost. If you are running a 70B target model and a 1B draft model, and the draft achieves a mean accepted length of 3 tokens, you have roughly tripled throughput for latency-sensitive traffic.
The mathematical guarantee that matters is that the output distribution is identical to the target model’s distribution. Speculative decoding is not an approximation. The rejection sampling ensures that tokens with high draft probability get accepted, tokens the target model would assign low probability to get rejected and replaced. Your quality does not degrade. This is what separates it from quantization, which genuinely changes the model’s output distribution; speculative decoding changes the inference algorithm, not the model.
The Evolution: From Speculative Sampling to EAGLE-3
The original speculative decoding paper from Google Research used a separate smaller model as the draft model. Simple, clean, effective in the right conditions. The problem was training and maintaining a high-quality draft model for every target model you wanted to accelerate. If you served a dozen different models, you needed a dozen different draft models, each trained to produce high-acceptance-rate drafts for its corresponding target.
Medusa changed the model architecture by attaching parallel prediction heads directly to the target model. Instead of a separate draft model, Medusa adds K independent heads that each predict a token at a different offset (head 1 predicts t+1, head 2 predicts t+2, and so on). The heads share the target model’s hidden states but do not condition on each other. This keeps memory overhead tiny, under one percent of the target model size, and eliminates the separate draft model entirely. The tradeoff is independence: head 2 predicts t+2 without knowing what head 1 predicted for t+1, which hurts acceptance rates on tasks where token choices have strong sequential dependencies (code, structured output, mathematical reasoning).
EAGLE took a different approach. Instead of parallel independent heads, EAGLE trains a small autoregressive draft model that operates on the target model’s internal feature vectors rather than the output token embeddings. The draft model learns to predict the target model’s future feature states, and each draft token is conditioned on the previous draft token’s prediction. This recaptures the sequential dependency that Medusa loses, and acceptance rates improve significantly, reaching 85-95% on instruction-following tasks versus Medusa’s 60-80%.
EAGLE-3 (released in early 2026 and merged into vLLM, SGLang, and TensorRT-LLM main) makes two key changes over EAGLE-2. First, the draft head receives fused context from multiple transformer layers, not just the final layer’s hidden states. This multi-layer fusion gives the draft head richer signal about what the target model “knows” at the current position, which pushes acceptance rates higher, particularly on coding and reasoning tasks where early-layer representations carry useful syntactic and semantic information. Second, EAGLE-3 introduces dynamic speculation length: rather than always proposing a fixed K candidates, the draft head estimates confidence and adjusts K per step, avoiding wasteful drafts when the model is in a high-entropy state.
EAGLE 3.1 (May 2026, shipping in vLLM v0.22.0) adds one important fix for production deployments: it corrects an attention drift bug where the draft head, under long contexts or unusual chat templates, gradually shifts attention toward its own generated tokens and away from the original prompt. The result is that EAGLE-3 acceptance rates can degrade on long-context workloads without 3.1. If you are running context lengths above 8K tokens in production, EAGLE 3.1 is the version to deploy, not vanilla EAGLE-3.
P-EAGLE, described in the vLLM team’s March 2026 blog post, parallelizes the EAGLE draft generation step itself. Standard EAGLE generates draft tokens autoregressively, which means K sequential draft steps before the target model verification pass. P-EAGLE restructures the draft generation to run multiple draft steps in parallel, reducing the serial latency of draft generation at the cost of some acceptance rate. On B200 GPUs, the vLLM team reports P-EAGLE achieves 1.05x to 1.69x speedup over vanilla EAGLE-3 on their benchmark suite. The speedup is larger when the draft model is the bottleneck (which happens more on newer, faster target model hardware).
The Concurrency Cliff You Will Hit in Production
Here is the thing that kills speculative decoding deployments: it is a latency optimization, not a throughput optimization at scale. Understanding this distinction is the difference between a successful deployment and a confused postmortem.
Speculative decoding helps when your GPU is memory-bandwidth-bound. At low concurrency (batch size 1-8), a single inference request needs the GPU to load all model weights from HBM for each token, and the compute units finish their arithmetic before the next weight chunk arrives. The GPU is underutilized. Speculative decoding exploits this underutilization: the draft model’s compute is small enough that it fits in the gaps, and the verification pass batches the target model’s work, making better use of each weight-loading cycle.
At high concurrency (batch size 32+), the GPU is compute-bound. Every compute unit is busy. Adding draft model forward passes does not fit into idle cycles; it competes with ongoing target model work. The expected speedup drops toward 1.0x, and depending on how your inference framework schedules things, you can see throughput decrease.
This means the decision about whether to enable speculative decoding depends on your serving profile, not just your hardware. Deployment guidance from vLLM and BentoML’s inference handbook is consistent: disable speculative decoding above batch size 32, and re-evaluate around batch size 16-24 depending on draft model size. If you are running a high-QPS API endpoint with many concurrent users, you are often better off without it. If you are running a low-QPS endpoint where individual request latency drives user experience (interactive chat, coding assistants, voice AI pipelines), speculative decoding is the right tool.
I run into this constantly. A team builds a beautiful EAGLE-3 deployment, benchmarks it at batch size 1, sees 3x speedup, deploys it to production, and then wonders why their P95 latency looks identical to before while their GPU memory headroom has shrunk. The answer is almost always that production traffic is running at batch size 20-40 during peak hours, and speculative decoding is helping during off-peak periods only.
The right monitoring approach is to track speculative decoding acceptance rate and mean accepted length alongside your usual inference metrics. If mean accepted length is 3.0 at batch size 4 and drops to 1.2 at batch size 24, that tells you speculative decoding is still accepting tokens at the same rate but the absolute speedup has collapsed because the verification pass is no longer cheap. Some teams implement adaptive logic that disables speculative decoding when observed batch size exceeds a threshold; vLLM has configuration flags to help with this.
Medusa vs EAGLE-3: Which One to Deploy
For most teams in 2026, EAGLE-3 is the right choice. The acceptance rate advantage over Medusa is real and persistent across workload types, and NVIDIA publishes pre-trained EAGLE-3 heads for the most popular open models (Llama 3.1, Llama 3.3, Qwen, Mistral, and others) through the TensorRT-LLM model collection and HuggingFace’s speculative decoding hub. If your target model has a published EAGLE-3 head, you can enable speculative decoding in about thirty minutes without training anything.
Medusa earns its place in specific circumstances. If memory is genuinely constrained (you are maxing out GPU memory headroom and cannot fit even a small separate draft model), Medusa’s sub-1% overhead versus EAGLE-3’s 1-2B parameter draft model can make the difference between fitting and not fitting. If you are running a highly customized or fine-tuned target model for which no pre-trained EAGLE-3 head exists and you do not want to invest training compute, Medusa’s heads can be added and trained in a fraction of the time it takes to train a full EAGLE-3 draft model. And if your workload is very diverse (many different prompt domains), Medusa’s simpler architecture sometimes shows more consistent acceptance rates than EAGLE-3, which can be sensitive to domain mismatch between its training distribution and your production prompts.
That last point is worth emphasizing. An EAGLE-3 head trained on general instruction-following data will achieve its published acceptance rates on general instruction-following tasks. If you are running a specialized code generation workload, you may get better results from an EAGLE-3 head fine-tuned on code (GitHub Copilot-adjacent datasets), or from Medusa with heads tuned on code. The acceptance rate is what drives speedup, and acceptance rate is directly tied to how well your draft model understands your prompt distribution.
Framework Configuration
If you are running vLLM (which you probably are, given its dominance as of late 2026), EAGLE-3 configuration is straightforward. Set --speculative-model to the path of your EAGLE-3 draft model (or HuggingFace repo), --num-speculative-tokens to 5 as a starting point, and --speculative-eagle-topk to 8. The vLLM docs note that going above 8 for topk rarely improves throughput because the expected accepted tokens per step plateaus, and going below 3 for speculative tokens leaves throughput on the table. You will also want to set --speculative-disable-by-batch-size or equivalent to your computed concurrency threshold so speculative decoding disengages automatically at high load.
# vLLM EAGLE-3 launch configuration (replace with your actual model and draft model paths)
vllm serve meta-llama/Llama-3.1-70B-Instruct \
--speculative-model your-eagle3-draft-model-path \
--num-speculative-tokens 5 \
--speculative-eagle-topk 8 \
--max-model-len 8192
For SGLang, the RadixAttention KV cache is fully compatible with EAGLE-3 draft verification, which is a meaningful advantage. SGLang’s KV cache reuse across requests (the core of its performance advantage over vLLM in prefix-heavy workloads) continues to work with speculative decoding enabled. Configuration follows similar parameters to vLLM. If you are already using SGLang for its prefix cache performance (as covered in the prefix caching guide), you can stack speculative decoding on top without losing the KV cache benefits.
TensorRT-LLM’s speculative decoding support is the most mature at the CUDA kernel level. NVIDIA’s TensorRT-LLM team publishes EAGLE-3 heads for their supported model list, and the integration is tested end-to-end on their hardware. If your serving stack is already TensorRT-LLM-based (common in enterprise deployments on DGX or HGX systems), this is the path of least resistance. The tradeoff is TensorRT-LLM’s engine compilation overhead: you need to compile separate engines for the draft and target models, and engine compilation time and disk footprint both increase.
Draft Model Training and Selection
If a pre-trained EAGLE-3 head does not exist for your model, you have two options: train one or use a separate smaller model as the draft.
Training an EAGLE-3 head is more tractable than training a full model. The draft head is a small autoregressive transformer (typically 1-4B parameters) trained to predict the target model’s hidden-state continuations. You need access to the target model’s internal activations during training, which means you need either the full model weights or a cooperating serving endpoint. Training data requirements are modest relative to pre-training: the draft head needs to learn your target model’s specific prediction patterns, so you want representative data from your target domain and enough compute to converge the draft head’s parameters (not the target model itself). Research frameworks like SpecForge (described in a March 2026 arxiv paper) describe infrastructure for this training loop, and open-source implementations continue to emerge as the technique matures.
Using a generic smaller model from the same family (for example, using Llama 3.1-8B as a draft for Llama 3.1-70B) is simpler but usually underperforms a purpose-trained EAGLE-3 head. The acceptance rates with family-draft approaches typically land in the 0.55-0.70 range versus 0.80-0.88 for well-trained EAGLE-3 heads on matched domains. Whether the setup simplicity justifies the performance gap depends on how much your team wants to invest in the draft model pipeline.
For teams building on top of GPU cloud infrastructure without direct model access, the pre-trained EAGLE-3 heads published by NVIDIA and the community are the practical starting point. Check the NVIDIA TensorRT-LLM Speculative Decoding Modules collection and HuggingFace’s speculative-decoding tagged models before deciding you need to train your own.
Production Monitoring for Speculative Decoding
The metrics that matter for speculative decoding are not the same as standard inference metrics. Tracking only TTFT (time-to-first-token) and TPS (tokens-per-second) will not tell you whether speculative decoding is actually helping.
Add these to your observability stack (whichever of Langfuse, Arize, or native vLLM metrics you are using per your LLM observability setup):
Mean accepted length (MAL): average number of draft tokens accepted per speculative step. Target above 2.5 for meaningful speedup. Below 1.5 and speculative decoding is adding overhead with minimal gain.
Draft acceptance rate: fraction of draft tokens accepted. Should be above 0.7 for EAGLE-3 on matched-domain workloads.
Speculative decoding overhead: the latency added by draft generation per request, separate from target model latency. If this number grows relative to total request latency as concurrency increases, you are hitting the compute-bound regime.
Per-batch-size MAL: track MAL broken down by the batch size at the time of the request. This is how you find your specific concurrency cliff, rather than trusting the generic guidance of “batch size 32.”
I run these metrics in a Prometheus scrape from vLLM’s /metrics endpoint, which exposes vllm:spec_decode_draft_acceptance_rate and vllm:spec_decode_num_accepted_tokens (divided by num_draft_tokens to get MAL). Alert on MAL falling below 2.0 for more than five minutes during peak traffic; that is usually a sign that something changed in the traffic distribution or the draft model is mismatched to the request types currently flowing in.
When to Skip Speculative Decoding Entirely
I want to be honest about cases where speculative decoding is not the right lever. It is not a universal inference optimizer, and pushing it into situations where it does not fit creates operational complexity without the performance payoff.
Skip it if your serving pattern is dominated by long-output requests at high concurrency. A coding assistant generating 4,000-token diffs for 100 simultaneous users is compute-bound. Speculative decoding will not help, and the draft model adds GPU memory pressure that could otherwise hold more KV cache.
Skip it if your primary bottleneck is prefill latency, not decode latency. Speculative decoding accelerates the decode phase (the token-by-token generation after the prompt is processed). If your users are waiting on prompt processing (long system prompts, long documents in the context), prefix caching and disaggregated prefill (as covered in the llm-d guide) are the right tools, not speculative decoding.
Skip it if you are already using quantization aggressively. FP8 quantization changes the target model’s computation in ways that can reduce draft model acceptance rates, particularly if the draft model was trained against a full-precision or BF16 target. You can still combine them, but you may need to retrain the draft head against the quantized target to recover acceptance rates.
And skip it if your model is a mixture-of-experts architecture with very sparse activation patterns. MoE models present a different memory access profile (only active expert weights are loaded per token), which changes the memory-bandwidth-bound calculation that speculative decoding exploits. The gains are more variable and less predictable than with dense models.
The Broader Inference Optimization Stack
Speculative decoding fits into a broader inference optimization hierarchy. I think of it in layers: hardware selection and sizing at the bottom, then quantization to reduce model footprint, then prefix caching to avoid recomputing shared prompt prefixes, then speculative decoding to maximize decode throughput for latency-sensitive traffic, then request batching and scheduling at the top.
Each layer compounds with the others when implemented correctly, but each also has its constraints. Quantization narrows the set of hardware that can run the model efficiently. Prefix caching requires predictable prompt structure. Speculative decoding requires low concurrency and a matched draft model. Understanding where each technique applies is what separates effective AI FinOps from throwing optimizations at the wall.
The teams doing this well in 2026 are the ones who have built serving pipelines that apply different optimization profiles to different request types. A coding assistant request at 3am with no concurrent traffic gets speculative decoding at 5 draft tokens per step. The same model serving 200 concurrent API calls at peak gets standard autoregressive decoding with continuous batching. The switch happens automatically based on observed batch size and the acceptance rate metrics described above.
This is not theoretical; it is the operational maturity that distinguishes teams paying reasonable inference bills from teams who are permanently surprised by their GPU costs. If you are evaluating your complete LLM inference infrastructure, speculative decoding belongs in the evaluation alongside the inference engine comparison and the hardware provisioning decisions.
What Is Coming Next
The speculative decoding research pipeline is active in 2026. Batch speculative decoding (optimizing verification pass efficiency at moderate batch sizes to push the concurrency cliff higher) is an active area, with arxiv preprints showing meaningful improvement at batch sizes 8-16. Multi-draft speculative decoding (maintaining a tree of candidate drafts rather than a single chain) is seeing production experiments at some of the larger serving providers.
P-EAGLE’s parallel draft generation is also an interesting direction because it directly addresses the draft latency that becomes the bottleneck on fast hardware. As target model forward pass speed improves (B200 and the next generation), the draft model’s serial generation becomes a larger fraction of total inference time. Parallelizing draft generation is the natural response.
The practical takeaway for teams building now: enable EAGLE-3 for your latency-sensitive endpoints, measure acceptance rate and MAL carefully, set a batch-size ceiling for automatic disengagement, and revisit the configuration quarterly as both the models and the frameworks evolve quickly. This is not a set-and-forget optimization; it is more like cache tuning, where you get the configuration right for your workload profile and then monitor for drift.
For teams already thinking about how speculative decoding interacts with disaggregated inference architectures (separate prefill and decode nodes), the llm-d overview covers how the Kubernetes-native disaggregated inference approach handles KV cache routing, which is the complementary piece of the puzzle on the memory side.
Twenty years of infrastructure work has taught me that the best optimizations are the ones that are invisible to users and cheap to maintain for operators. Speculative decoding, when tuned correctly for your workload, is one of those rare interventions that actually earns that description.
Get Cloud Architecture Insights
Practical deep dives on infrastructure, security, and scaling. No spam, no fluff.
By subscribing, you agree to receive emails. Unsubscribe anytime.