Speculative Decoding: Faster LLMs With a Tiny Sidekick

Your GPU Is Sitting Idle Between Tokens — Here’s How to Fix That

LLM inference is embarrassingly serial. The big model generates one token, ships it, waits, generates the next one. Your RTX 3090 is mostly cooling its heels while the attention mechanism churns. That’s not a hardware problem — it’s an architectural one.

Speculative decoding is the fix. A tiny sidekick model sprints ahead, drafts a handful of tokens at once, and the big model verifies the whole batch in a single forward pass. If the draft was right, you just got 4-5 tokens for the price of 1. If it was wrong, you pay the normal cost and move on. Either way, your throughput jumps 2-3x without touching model quality.

It’s like the passing lane on a highway. A fast car (draft model) zips ahead and scouts the road. The truck behind it (target model) either follows that path or takes a different exit — but the truck never actually has to drive slowly in the right lane to figure out where it’s going. And in 2026 there’s a whole fleet of scout cars to choose from — separate draft models, trained drafter heads baked into the model itself, confidence-scored schedulers — all playing the same passing-lane game with different tradeoffs.

How It Actually Works

Here’s the core loop:

1. Draft model generates k tokens speculatively (typically 4-8)
2. Target model runs a single forward pass over those k tokens in parallel
3. Verification: for each draft token, compare the target model’s probability distribution at that position. Accept if they agree (within a threshold), reject the first disagreement and everything after it
4. Target model generates the correction token at the first rejection point
5. Repeat

The math works out because the target model’s forward pass over k tokens is much cheaper than k sequential forward passes. GPU parallelism is doing actual work instead of waiting for autoregressive step n to finish before starting n+1.

The guarantee: output quality is identical to just running the target model. Speculative decoding is lossless — you’re not approximating anything, just reordering the compute.

Pairing Rules (Or: You Can’t Just Grab Any Two Models)

The draft and target models need to be compatible:

• Same tokenizer: if the tokenizer differs, token IDs don’t map to the same vocabulary, and verification breaks completely
• Same model family (strongly preferred): a 3B Llama 3.1 draft + 70B Llama 3.1 target works great. A 3B Mistral draft + 70B Llama 3.1 target will have poor acceptance rates and you’ll barely see any speedup
• Draft model should be 5-10x smaller: draft too small and accuracy tanks, draft too close in size and the VRAM overhead isn’t worth it

Good pairings that actually work in 2026:

• llama3.1:8b draft + llama3.1:70b target
• deepseek-r1:7b draft + deepseek-r1:70b target
• qwen2.5:3b draft + qwen2.5:72b target
• phi-3-mini draft + phi-3-medium target (same family matters more than size ratio here)

llama.cpp: The --draft Flags

llama.cpp has had speculative decoding since late 2023. The llama-speculative binary (previously speculative) is what you want.

# Basic speculative decoding run
./llama-speculative \
  -m models/llama-3.1-70b-instruct.Q4_K_M.gguf \
  --model-draft models/llama-3.1-8b-instruct.Q4_K_M.gguf \
  -p "Explain how speculative decoding works" \
  -n 512 \
  --draft 8 \
  --draft-max 16 \
  -t 8 \
  --color

Key flags:

• --model-draft — path to the draft model GGUF
• --draft — number of tokens to speculatively draft per step (start at 8)
• --draft-max — max draft length (tune this up if acceptance rate is high)
• -t — threads (set to physical core count, not logical)

Measuring Acceptance Rate

llama.cpp logs draft acceptance stats when you use --verbose:

./llama-speculative \
  -m models/llama-3.1-70b.Q4_K_M.gguf \
  --model-draft models/llama-3.1-8b.Q4_K_M.gguf \
  -p "Write a Python script to parse JSON" \
  -n 256 \
  --draft 8 \
  --verbose 2>&1 | grep -E "(draft|accept|tokens/s)"

You’re looking for acceptance rates above 70% to see meaningful speedup. Below 50% and you might as well skip the draft model entirely — the verification overhead starts costing more than it saves.

Quick Benchmark Script

import subprocess
import time
import re

PROMPT = "Write a detailed explanation of transformer attention mechanisms"
MODELS = {
    "baseline": {
        "cmd": ["./llama-cli", "-m", "models/llama-3.1-70b.Q4_K_M.gguf",
                "-p", PROMPT, "-n", "256", "--no-display-prompt"]
    },
    "speculative": {
        "cmd": ["./llama-speculative",
                "-m", "models/llama-3.1-70b.Q4_K_M.gguf",
                "--model-draft", "models/llama-3.1-8b.Q4_K_M.gguf",
                "-p", PROMPT, "-n", "256", "--draft", "8", "--no-display-prompt"]
    }
}

for name, config in MODELS.items():
    start = time.time()
    result = subprocess.run(config["cmd"], capture_output=True, text=True)
    elapsed = time.time() - start

    # llama.cpp reports tok/s in stderr
    match = re.search(r"eval time.*?(\d+\.\d+) tokens per second", result.stderr)
    tps = match.group(1) if match else "N/A"

    print(f"{name}: {tps} tok/s ({elapsed:.1f}s total)")

Real-world numbers on a 3090 (24GB VRAM) with Q4_K_M quantization:

• Baseline 70B: ~12 tok/s
• Speculative (8B draft): ~28-32 tok/s
• Acceptance rate: ~75-80% on code generation tasks

That’s a 2.3-2.7x speedup on a task where the draft model is reasonably aligned.

vLLM: speculative_config

If you’re running vLLM for serving (OpenAI-compatible API endpoint, batched requests), speculative decoding is a config block away since vLLM 0.4.x.

from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3.1-70B-Instruct",
    speculative_config={
        "model": "meta-llama/Llama-3.1-8B-Instruct",
        "num_speculative_tokens": 5,
    },
    tensor_parallel_size=2,  # if you have multiple GPUs
    dtype="bfloat16",
    max_model_len=4096,
)

sampling_params = SamplingParams(temperature=0.7, top_p=0.9, max_tokens=512)
outputs = llm.generate(["Explain Kubernetes networking in plain English"], sampling_params)
print(outputs[0].outputs[0].text)

Or via the CLI for serving:

vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --speculative-config '{"model": "meta-llama/Llama-3.1-8B-Instruct", "num_speculative_tokens": 5, "draft_tensor_parallel_size": 1}' \
  --tensor-parallel-size 2 \
  --port 8000

The standalone --speculative-model and --num-speculative-tokens flags were deprecated a while back — everything now goes into the single --speculative-config JSON blob. The draft_tensor_parallel_size: 1 key keeps the small draft model on a single GPU while the big model splits across both — which is usually what you want. Running the 8B draft across 2 GPUs is overkill and just adds NVLink overhead.

vLLM Batch Dynamics Warning

Here’s the thing about vLLM and speculative decoding: it shines for low-concurrency or single-request scenarios. Under heavy batch load (16+ concurrent requests), the batch dynamics get complicated. Requests are at different positions in their draft/verify cycle, acceptance rates across diverse prompts are lower, and you can actually see worse throughput than baseline. Check your vllm_metrics Prometheus endpoint after deploying — if average acceptance rate drops below 60%, you’re likely better off without it for that traffic pattern.

Ollama: 2026 Status

Honest answer: Ollama’s speculative decoding support is still incomplete as of mid-2026. The upstream llama.cpp integration is there in theory, but the Ollama model management layer doesn’t expose a clean way to declare draft model pairings in a Modelfile.

There’s been a GitHub issue tracking this for a while. The workaround is to drop down to llama-speculative directly (llama.cpp binary) if you need speculative decoding on local hardware — Ollama wraps llama.cpp anyway, you’re just bypassing the friendly abstraction layer.

If you’re primarily using Ollama for its API compatibility and model library convenience, this is one area where you’ll want to reach past it. Your 2 AM self will appreciate knowing this before you spend an hour trying to figure out why the draft parameter in the Ollama API isn’t doing anything.

When Speculative Decoding Doesn’t Help (Read This Before Getting Excited)

Speculative decoding is not free lunch. There are real cases where it’s neutral or harmful:

Greedy / Low Temperature Sampling

With temperature=0 (greedy decoding), the target model is deterministic — it will pick the same token every time. The draft model often aligns well here, so acceptance rates are high… but the speedup depends heavily on whether draft and target are well-matched. For highly deterministic tasks like structured JSON generation with constrained grammars, you might see less benefit because the “easy” tokens were already fast.

Very Short Outputs

If you’re generating 5-10 tokens, the overhead of loading the draft model into VRAM and setting up the speculative pipeline costs more than you gain. The breakeven point is roughly 50-100 output tokens.

VRAM Budget

The draft model lives in VRAM alongside the target. An 8B draft in Q4 is ~4-5GB. If you’re already running a 70B model in Q4 at 35-38GB on a 40GB A100, that’s a tight squeeze. OOM mid-generation is not a fun experience. Quantize the draft more aggressively (Q2_K is fine, it just needs to be directionally correct, not perfect).

Diverse / Creative Prompts

Code generation and factual Q&A have high acceptance rates because the “right” token is often predictable. Open-ended creative writing, roleplay, or highly stochastic tasks have lower acceptance rates because the target model’s distribution diverges from the draft model’s. You might still see 1.3-1.5x speedup, but not the 2-3x you’d hope for.

The Whole Family: MTP, EAGLE, Medusa, DSpark & Friends

Here’s the thing nobody tells you upfront: speculative decoding isn’t one trick, it’s a whole family tree. Every variant below is doing the exact same core move — draft cheap, verify in parallel, guarantee lossless output — they just disagree on whether the drafter is a separate model, a trained head bolted onto the target, or no extra model at all. That choice drives everything else: how much VRAM it costs, how hard it is to set up, and whether you need a special checkpoint.

Gemma 4 MTP (Multi-Token Prediction) — Google shipped this with Gemma 4 in May 2026, and it’s the most accessible “next-gen” option for a home lab right now. Instead of pairing your target model with an entirely separate draft model, Gemma 4 ships with tiny companion “drafter” checkpoints — about four layers deep, a few hundred MB — that ride alongside the ~31B target. It’s still self-drafting speculative decoding under the hood: the lightweight head proposes several tokens ahead, the big model verifies them in one forward pass, output is bit-for-bit identical to running the target alone. Because the drafter shares the target’s training lineage instead of being a bolted-on unrelated model, it directly attacks the memory-bandwidth bottleneck — the real cost of autoregressive decoding is shuttling weights between VRAM and compute for every single token, and skipping that shuttle for accepted tokens is where the speedup comes from. Real numbers: up to ~3x speedup, more realistically 1.8-3x depending on hardware and prompt shape. The community’s already benchmarked it — around 140 tok/s on a 12GB RTX 4070 Super, roughly 2x on dual 3090s. It’s supported in Hugging Face Transformers and llama.cpp today, so if you’re already running Gemma 4 there’s no separate model to hunt down — the drafter ships with it.

DeepSeek DSpark — open-sourced by DeepSeek and Peking University in June 2026 under MIT license, DSpark is aimed at a different problem: serving at scale, not a single home GPU. It’s a semi-autoregressive speculative decoding framework — pairs a small semi-autoregressive drafter with a confidence head and a hardware-aware scheduler. The clever bit is “confidence-scheduled validation”: instead of verifying every drafted token with equal rigor, it decides what not to verify, skipping full verification of high-confidence spans. The semi-autoregressive drafting also sidesteps “suffix decay” — the tendency of draft quality to degrade toward the tail end of a long drafted span, which plagues naive autoregressive drafters. On DeepSeek-V4-Flash it’s 60-85% faster per-user; on V4-Pro, 57-78% faster; aggregate throughput gains run up to ~6.6x. Still lossless — output matches the target byte-for-byte. It ships with a companion training framework, DeepSpec, for building your own drafters. If you’re running a single 70B model on one GPU, this isn’t really your lane; if you’re serving DeepSeek-V4 to real traffic, it’s the biggest lever on this list.

EAGLE (Extrapolation Algorithm for Greater Language-model Efficiency) — trains a lightweight autoregressive head directly on the target model’s feature space instead of using a separate model. Because it shares the base model’s hidden states, acceptance rates hit 85-90%+ vs. 70-80% for a separate draft model. EAGLE-2 and EAGLE-3 push this further. Several llama.cpp forks support it; mainstream merge is in progress.

Medusa — adds multiple “medusa heads” to the base model, each predicting tokens at offset +1, +2, +3, etc., in parallel. No separate model, but the heads need to be fine-tuned on top of the frozen base — you can’t just drop Medusa onto any checkpoint.

Lookahead Decoding — uses Jacobi iteration to maintain a 2D window of “guesses” and verifies multiple token paths at once. No extra model, no fine-tuning, but the implementation is fiddlier and the speedup tops out around 1.5-2x. Good for when you genuinely can’t spare a byte of VRAM.

Which One Should You Actually Use?

All of these are lossless, so this is a cost-and-setup-effort decision, not a quality one. For a single home GPU today, a classic matched draft model or Gemma 4 MTP (if you’re already on Gemma) is the sweet spot — easy, no exotic checkpoints, real speedup. Reach for EAGLE if you can get your hands on a compatible checkpoint and want the highest acceptance rate. DSpark is the one to watch if you’re serving DeepSeek-V4 in production and throughput is the metric that matters.

The Practical Summary

If you’re running a 70B class model locally and want faster inference:

1. Grab the matching smaller model from the same family (8B for Llama, 7B for Qwen/DeepSeek)
2. Use llama-speculative with --draft 8, tune up if acceptance rate is above 80%
3. For vLLM serving, add speculative_config and monitor acceptance rate under real traffic
4. Skip it if you’re VRAM-constrained, generating very short outputs, or on Ollama (for now)
5. On Gemma 4? Try its built-in MTP drafter first — no extra model to source. Serving DeepSeek-V4 at scale? Watch DSpark; that’s where the throughput gains live

Your GPU already paid for itself. Make it earn its keep.

Related Reading

• Gemma 4 vs Qwen3.6
• Mixture of Experts (MoE) for Self-Hosters, Demystified
• KV Cache Quantization: Free LLM Context, Almost
• API vs Self-Hosted LLMs: The Real Cost
• Continue.dev vs Cody vs Tabby: AI Code Help Without the Cloud