What is Speculative Decoding in vLLM

Speculative decoding is a technique to speed up LLM inference without changing the output distribution. It uses a fast draft model to propose tokens and a slow target model to verify them. This post explains how it works, what the AMD benchmark results reveal, and how to configure it in vLLM.

Key Takeaways

Speculative decoding keeps the target model responsible for output; a lightweight draft component proposes tokens and the target verifies them in one pass.
Peak measured throughput on AMD hardware was 2.87x (DFlash on gemma-4-26B-A4B-it, MATH500), but some combinations fell below baseline.
Proposal length is not constant: the best num_speculative_tokens ranged from N=4 to N=7 depending on the model, method, and dataset.
Native MTP needs no separate checkpoint; EAGLE-3, DFlash, and DSpark load additional draft weights and require memory headroom.

How Draft-and-Verify Works

Standard autoregressive decoding commits one token per model step. Generating four output tokens means four sequential decode passes, each one appending the new token and re-running the model. During long generations, that token-by-token loop dominates latency and limits serving throughput.

Speculative decoding divides the work into two parts: draft and verify. A lightweight draft component proposes several candidate future tokens. The target model then checks those candidates in a single verification pass, left to right. Accepted tokens are committed; at the first rejection, the target provides its own replacement token and the remaining candidates are discarded.

Five Drafting Methods, Three Architectures

Every speculative method follows the same draft-and-verify loop, but they differ in what the draft component is and how it receives information from the target model. vLLM’s exploration groups them into three architectural categories.

Native MTP (Multi-Token Prediction)

This method is built directly into the target model’s architecture as an auxiliary prediction path. It combines the hidden representation from the target model with the current token embedding to predict the first draft token, then reuses the MTP path sequentially for subsequent tokens. No separate checkpoint is loaded, and parts of the MTP path often share components with the target model, keeping memory overhead modest.

Gemma 4 MTP

This is a separately packaged draft component paired with a specific target model. It consumes the target’s activations and shares the target’s KV cache, so it reuses context the target has already computed rather than processing the accepted prefix independently.

Dedicated Target-Conditioned Draft Networks

These are separate speculator models trained for a specific target.

EAGLE-3 records hidden states from the beginning, middle, and end of the target’s forward pass, fuses them into one feature, and drafts autoregressively.
DFlash predicts an entire block of future positions in parallel using an anchor token and masked positions, supplying fused target context as additional Key/Value information in every draft layer.
DSpark extends DFlash with a lightweight Markov head that adds sequential dependence between tokens and a confidence head for prefix selection (the confidence head was not active in vLLM’s tested path).

Method	Draft component	Separate checkpoint	Token generation
Native MTP	Model-native auxiliary path	No	Sequential
Gemma 4 MTP	Paired MTP drafter, shares KV cache	Yes	Sequential
EAGLE-3	Autoregressive draft network	Yes	Sequential
DFlash	Parallel draft network	Yes	Parallel (one pass)
DSpark	DFlash backbone + Markov head	Yes	Parallel + sequential correction

Pretrained draft checkpoints are widely published. Google provides MTP assistants for Gemma 4, Z-Lab maintains DFlash checkpoints, Red Hat AI has EAGLE-3, DFlash, and DSpark models, DeepSeek’s DeepSpec collection covers all three methods for Qwen3 targets, LightSeek focuses on EAGLE-based models for Kimi, and Inferact publishes draft models for MiniMax and Kimi.

What AMD Measurements Show

The vLLM team measured generated tokens per second against an autoregressive baseline across nine target models, sweeping the number of speculative tokens. The results, run on AMD Instinct MI300X and MI355X GPUs with ROCm, indicate that no single method performs best in all cases (source).

For gemma-4-26B-A4B-it, the largest throughput ratios were 2.74x and 2.62x for Gemma 4 MTP on GSM8K and MBPP, and 2.87x and 2.79x for DFlash on MATH500 and HumanEval. EAGLE-3 ranged from 2.11x to 2.27x across four datasets. On gemma-4-31B-it, Gemma 4 MTP reached 2.00x on GSM8K and 1.99x on MBPP, while DFlash hit 2.34x on MATH500 and 2.05x on HumanEval.

The picture changes sharply on smaller models. For Qwen3-8B, DSpark ranged from 1.15x on MATH500 to 1.63x on GSM8K, DFlash from 1.08x to 1.27x, and EAGLE-3’s largest MATH500 measurement actually fell below baseline. On larger Qwen3.5 and Qwen3.6 models, native MTP beat DFlash, with the largest ratio of 2.20x for Qwen3.5-122B-A10B on MATH500.

The MiniMax-M3-MXFP8 target reached 2.09x with EAGLE-3 on HumanEval at N=4. Kimi-K2.5 reached up to 2.33x with EAGLE-3 and 2.68x with DFlash. For sequential methods, throughput often rose over the first few values of N before leveling off; for DFlash and DSpark, N=7 was frequently among the higher-throughput settings while larger values did not consistently improve results. The proposal length tied to the best result varied across cases.

Enabling Speculative Decoding in vLLM

Speculative decoding is configured through --speculative-config. The supported method values are mtp, eagle3, dflash, and dspark. Native MTP omits the model field because the draft component ships with the target model:

vllm serve Qwen/Qwen3.5-27B \
 --speculative-config '{
 "method": "mtp",
 "num_speculative_tokens": 5
 }'

# Note: num_speculative_tokens must be compatible with the checkpoint's
# prediction depth; a larger value reuses the MTP path in extra forward
# passes, adding sequential drafting work before verification.

For Gemma 4 MTP, EAGLE-3, DFlash, and DSpark, the model field points to a checkpoint trained for a specific target:

vllm serve Qwen/Qwen3-8B \
 --speculative-config '{
 "method": "dflash",
 "model": "z-lab/Qwen3-8B-DFlash",
 "num_speculative_tokens": 7
 }'

# Note: the draft checkpoint must match the target model and method, and
# the model card must support your hardware and inference backend. Verify
# before enabling rather than assuming compatibility.

Before turning any method on, confirm that the installed vLLM version supports the method and model architecture, that the draft checkpoint is compatible with the target, and that num_speculative_tokens is compatible with the checkpoint. Native MTP does not load a separate draft checkpoint and may share components like the embedding table or output head with the target. Gemma 4 MTP, EAGLE-3, DFlash, and DSpark all load additional draft weights, so reserve GPU memory headroom; the actual overhead depends on draft size, numerical precision, tensor-parallel configuration, and runtime buffers.

# Verify your vLLM build and available ROCm devices before serving
python -c "import vllm; print(vllm.__version__)"
rocm-smi --showproductname --showmeminfo vram

# Note: production use should pin the vLLM version and draft checkpoint
# revision, and A/B test the speculative config against your own eval set
# before rollout. The commands above do not test concurrency or long-context
# memory pressure.

Tuning, Memory, and Observability

The core tuning question is whether additional drafting work improves end-to-end serving performance. Acceptance behavior depends on the structure and predictability of real outputs, which is why the vLLM team used task-grounded benchmarks (GSM8K, MBPP, MATH500, HumanEval) rather than random token sequences. Random sequences would inflate apparent acceptance rates in ways that do not reflect actual model behavior.

Two indicators matter most: output-token throughput and speedup over the non-speculative baseline, plus mean accepted length and draft-token acceptance rate where available. The sweep results make one thing clear: you cannot pick a proposal length once and assume it holds. For sequential methods, throughput rises over the first few N values and plateaus; pushing N higher than the checkpoint’s useful depth adds sequential drafting work that reduces the gain. For parallel drafters like DFlash, longer blocks are more viable because all masked positions are predicted in one pass, but later positions are not conditioned on earlier sampled tokens within the same block, so effectiveness depends on the trained checkpoint and workload.

Memory is the other axis. Native MTP is the cheapest because it loads no separate checkpoint. Every other method adds draft weights to the GPU. On a model already sized close to the accelerator’s VRAM limit, that headroom requirement can force a smaller num_speculative_tokens or a different method entirely.

Limitations and Trade-offs

The headline numbers require the same skepticism applied to any vendor benchmark. These measurements come from vLLM’s own test environment and reflect specific hardware, software, target model, draft checkpoint, workload, and sweep configuration. The blog itself cautions that each result should be interpreted within its test setup, since model architecture, active parameter count, draft size, workload, and serving conditions all affect performance.

Independent research confirms that speculation is not uniformly beneficial. A 2024 study of over 350 experiments with LLaMA-65B and OPT-66B found that speedup depends heavily on draft-model latency, and that the draft model’s language-modeling capability does not correlate strongly with its speculative performance (source). A 2025 paper on “The Disparate Impacts of Speculative Decoding” showed speedup is not distributed evenly across tasks, consistently diminishing for under-fit and underrepresented tasks (source). Both point to the same practical conclusion: a draft method that performs well on GSM8K may do nothing, or even hurt, on your specific traffic.

The Qwen3-8B results illustrate this. EAGLE-3 was above baseline on GSM8K, HumanEval, and MBPP, but its largest MATH500 value fell below baseline. DSpark’s spread on the same model, 1.15x to 1.63x, means method choice is a per-workload decision, not a per-model one. If you deploy speculative decoding without benchmarking your own prompt distribution, you risk a regression.

There is also a memory-versus-speed trade-off that predates AMD specifically. Research on memory-constrained devices has shown speculative decoding uses extra memory allocations to generate candidates, and the acceptance rate drives the speedup. On low-memory GPUs, the same technique can slow down rather than speed up if draft weights crowd out KV-cache capacity. For teams already running near VRAM limits, the draft checkpoint’s footprint is the first thing to check.

For broader context on how vLLM fits into serving and how to benchmark engines without misleading yourself, see our comparison of local AI inference tools and 2026 inference engine guide. The same discipline applies here: measure at your real conversation length and concurrency, and treat a single tokens-per-second number as a starting point, not a verdict.

More in-depth coverage from this blog on closely related topics:

Sources and References

Sources cited while researching and writing this article: