Get Started

Capstone 14, Speculative-Decoding Inference Server

Speculative decoding, a cheap draft proposes tokens, the target model verifies them in one pass, is now a production-ready optimization, not a research trick. EAGLE-3 in vLLM 0.7 ships 2.5-3x throughput on real traffic. P-EAGLE (AWS 2026) pushed parallel speculation even further. SGLang's SpecForge trained draft heads at scale. Red Hat's Speculators hub published aligned drafts for common open models. TensorRT-LLM made speculative decoding first-class on NVIDIA. The 2026 production serving stack is vLLM or SGLang with EAGLE-family drafts, FP8 or INT4 quantization, and HPA on queue-wait. This capstone is to serve two open models at 2.5x+ baseline throughput with a full tail-latency report.

30 hoursCapstonePython (serving), C++ / CUDA (kernel inspection), YAML (configs)Free lesson

Problem

Speculative decoding became a commodity in 2026. EAGLE-3 draft heads train on the target model's hidden states and predict N tokens ahead; the target model verifies in a single pass. Acceptance rates of 60-80% translate to 2-3x end-to-end throughput. vLLM 0.7 integrates this natively. SGLang + SpecForge gives you the training pipeline. Red Hat's Speculators publishes aligned drafts for Llama 3.3 70B, Qwen3-Coder-30B MoE, GPT-OSS-120B.

The craft is in the serving operations, not the model. Acceptance rate drifts with the traffic distribution (ShareGPT vs code vs domain data). Tail latency under rejection is worse than without speculation, you must report p99 at multiple batch sizes, not just steady-state tokens/sec. Cost per 1M tokens vs Anthropic / OpenAI API is the credibility lever.

Concept

Speculative decoding has two layers. A draft model (EAGLE-3 head, ngram, or smaller target-aligned model) proposes k candidate tokens per step. The target model verifies all k in one pass; any prefix accepted replaces the greedy path. Acceptance rate depends on draft-target alignment and the input distribution.

EAGLE-3 beats ngram drafts on most traffic. P-EAGLE runs parallel speculation for deeper draft trees. The trade-off: P99 latency on rejection is higher because the verify pass is larger. The serving config must report batch-size-bucketed latency to surface this.

Deployment is Kubernetes. vLLM 0.7 runs one replica per GPU or tensor-parallel shard. HPA autoscales on queue-wait rather than CPU. FP8 (Marlin) and INT4 (AWQ) quants keep GPU memory inside an H100 / H200 envelope. The end-to-end report is throughput, acceptance rate, p50/p99 at batch 1/8/32, and $/1M tokens.

Architecture

Code
request ingress
    |
    v
vLLM server (0.7) or SGLang (0.4)
    |
    +-- draft: EAGLE-3 heads | P-EAGLE parallel | ngram fallback
    +-- target: Llama 3.3 70B | Qwen3-Coder-30B | GPT-OSS-120B
    |     quantized FP8-Marlin or INT4-AWQ
    |
    v
verify pass: batch k draft tokens through target
    |
    v (accept prefix; resample for rejected suffix)
    v
token stream back to client
    |
    v
Prometheus metrics: throughput, acceptance rate, queue wait, latency p50/p99
    |
    v
HPA on queue-wait metric

Stack

  • Serving: vLLM 0.7 or SGLang 0.4
  • Speculative methods: EAGLE-3 draft heads, P-EAGLE parallel speculation, ngram fallback
  • Draft training: SpecForge (SGLang) or Red Hat Speculators
  • Target models: Llama 3.3 70B, Qwen3-Coder-30B MoE, GPT-OSS-120B
  • Quantization: FP8 (Marlin), INT4 AWQ
  • Deployment: Kubernetes + NVIDIA device plugin; HPA on queue-wait metric
  • Eval: ShareGPT, MT-Bench-v2, GSM8K, HumanEval for domain-spread acceptance measurement
  • Reference: TensorRT-LLM speculative decoding for a vendor baseline

Build It

  1. Target model prep. Pick Llama 3.3 70B. Quantize to FP8 via Marlin. Deploy under vLLM 0.7 on 1xH100 (or 2x tensor-parallel).
  2. Draft source. Pull an aligned EAGLE-3 draft head from Red Hat Speculators (or train one via SpecForge). Load into vLLM's speculative-decoding config.
  3. Baseline numbers. Before speculation: tokens/s at batch 1/8/32, p50/p99 latency, GPU utilization. Publish.
  4. Enable EAGLE-3. Flip config; rerun the same benchmark. Report speedup, acceptance rate, p99 tail-latency delta.
  5. P-EAGLE. Enable parallel speculation; measure deeper draft tree vs serial EAGLE-3. Report the inflection where P-EAGLE helps vs hurts.
  6. Domain traffic. Run ShareGPT vs HumanEval vs domain-specific traffic through the same server. Measure acceptance rate per distribution. Identify when drafts drift.
  7. Second target model. Run the same pipeline on Qwen3-Coder-30B MoE. Draft is trickier (MoE routing noise). Report.
  8. K8s HPA. Deploy under K8s with HPA tracking queue_wait_ms. Demonstrate scale-out when load triples.
  9. Cost comparison. Compute $/1M tokens vs Anthropic Claude Sonnet 4.7 and OpenAI GPT-5.4 on the same eval. Publish.

Use It

Code
$ curl https://infer.example.com/v1/chat/completions -d '{"messages":[...]}'
[serve]     vLLM 0.7, Llama 3.3 70B FP8, EAGLE-3 active
[decode]    bs=8, accepted_tokens_per_step=3.2, acceptance_rate=0.76
[latency]   first-token 42ms, full-response 980ms (620 tokens)
[cost]      $0.34 per 1M output tokens at sustained throughput

Ship It

outputs/skill-inference-server.md describes the deliverable. A measured serving stack with speculative decoding, a full benchmark report, and a K8s deployment.

WeightCriterionHow it is measured
25Measured speedup vs baseline2.5x+ throughput at matched quality on two models
20Acceptance rate on realistic trafficPer-distribution acceptance-rate report
20P99 tail-latency disciplinep99 at batch 1/8/32 with and without speculation
20OpsK8s deploy, HPA on queue-wait, rollout smooth
15Write-up and methodologyClear explanation of what changed and why
100

Exercises

  1. Measure acceptance-rate degradation when the draft is one version behind the target (e.g., Llama 3.3 -> 3.4 drift). Build a monitoring alert.
  2. Implement ngram-fallback: if EAGLE-3 acceptance drops below a threshold, switch to ngram drafts. Report reliability improvement.
  3. Run a controlled MoE experiment: same Qwen3-Coder-30B with routing noise injected vs without. Measure draft acceptance sensitivity.
  4. Extend to H200 (141 GB). Report the model-size-per-replica headroom gained and whether you can serve an unquantized Llama 3.3 70B.
  5. Benchmark TensorRT-LLM speculative decoding on the same H100 hardware. Report where it wins vs vLLM.

Key Terms

TermWhat people sayWhat it actually means
Draft model"Speculator"Small model that proposes N tokens for the target to verify
EAGLE-3"2026 draft architecture"Draft head trained on target hidden states; ~75% acceptance
P-EAGLE"Parallel speculation"Tree of draft branches verified in one target pass
Acceptance rate"Hit rate"Fraction of drafted tokens accepted without resampling
Quantization"FP8 / INT4"Lower-precision weights to fit more model in GPU memory
Queue wait"HPA metric"Time a request waits in the pending queue before inference starts
Speculators hub"Aligned drafts"Red Hat Neural Magic hub of EAGLE drafts for common open models

Further Reading

Check what you learned

7 questions. Answer each one to see why.

Question 1 of 70 correct

What is the two-layer mechanism behind speculative decoding?

The full code

Every file for this lesson. Copy it, run it, then change it.

main.py
"""Speculative decoding server — draft/verify scheduler scaffold.

The hard architectural primitive is the draft/verify scheduler: a draft
model proposes k candidate tokens; the target model verifies them in one
batched pass; any accepted prefix is committed and the rejected suffix is
resampled from the target. This scaffold implements the scheduler with
synthetic token probabilities so the accept/reject logic and the throughput
math are observable end to end.

Run:  python main.py
"""

from __future__ import annotations

import random
from dataclasses import dataclass, field


# ---------------------------------------------------------------------------
# synthetic models  --  probability distributions over a tiny vocabulary
# ---------------------------------------------------------------------------

VOCAB = list("abcdefghij")


def softmax_from(seed: int) -> list[float]:
    rnd = random.Random(seed)
    weights = [rnd.random() for _ in VOCAB]
    total = sum(weights)
    return [w / total for w in weights]


def sample(dist: list[float], rng: random.Random) -> int:
    r = rng.random()
    acc = 0.0
    for i, p in enumerate(dist):
        acc += p
        if r <= acc:
            return i
    return len(dist) - 1


# ---------------------------------------------------------------------------
# target  --  the expensive model we are trying to save calls to
# ---------------------------------------------------------------------------

@dataclass
class TargetModel:
    calls: int = 0
    tokens_verified: int = 0

    def distribution(self, ctx_seed: int) -> list[float]:
        return softmax_from(ctx_seed * 7 + 13)

    def verify(self, draft_tokens: list[int], ctx_seed: int,
               rng: random.Random) -> tuple[list[int], int]:
        """Return (accepted_tokens, resampled_next). In one target call we can
        verify draft_tokens in a batched pass: the target produces a prob per
        position; we accept up to the first rejection."""
        self.calls += 1
        self.tokens_verified += len(draft_tokens) + 1
        accepted: list[int] = []
        for pos, tok in enumerate(draft_tokens):
            dist = self.distribution(ctx_seed + pos)
            # simple accept criterion: target prob on this token >= 0.5 * max prob
            if dist[tok] >= 0.5 * max(dist):
                accepted.append(tok)
            else:
                break
        # resample a next token from the target at the position after the accept
        ctx = ctx_seed + len(accepted)
        dist = self.distribution(ctx)
        next_tok = sample(dist, rng)
        return accepted, next_tok


# ---------------------------------------------------------------------------
# draft  --  a cheaper model that is mostly aligned with target
# ---------------------------------------------------------------------------

@dataclass
class DraftModel:
    calls: int = 0
    alignment: float = 0.80     # probability that draft picks what target would

    def propose(self, ctx_seed: int, k: int, rng: random.Random,
                target: TargetModel) -> list[int]:
        self.calls += 1
        draft_tokens: list[int] = []
        for pos in range(k):
            dist = target.distribution(ctx_seed + pos)
            # with prob alignment, emit target's best; otherwise sample a neighbour
            if rng.random() < self.alignment:
                draft_tokens.append(max(range(len(dist)), key=lambda i: dist[i]))
            else:
                draft_tokens.append(sample(dist, rng))
        return draft_tokens


# ---------------------------------------------------------------------------
# decode scheduler  --  speculative loop + baseline greedy for comparison
# ---------------------------------------------------------------------------

@dataclass
class Metrics:
    generated: int = 0
    target_calls: int = 0
    draft_calls: int = 0
    accepted_sum: int = 0

    def acceptance_rate(self, k: int) -> float:
        if self.target_calls == 0:
            return 0.0
        return self.accepted_sum / (self.target_calls * k)

    def tokens_per_target_call(self) -> float:
        return self.generated / max(1, self.target_calls)


def speculative_decode(n_tokens: int, k: int, rng: random.Random,
                       target: TargetModel, draft: DraftModel) -> Metrics:
    m = Metrics()
    ctx_seed = 1
    while m.generated < n_tokens:
        draft_tokens = draft.propose(ctx_seed, k, rng, target)
        m.draft_calls += 1
        accepted, next_tok = target.verify(draft_tokens, ctx_seed, rng)
        m.target_calls += 1
        m.accepted_sum += len(accepted)
        for tok in accepted:
            m.generated += 1
            ctx_seed += 1
            if m.generated >= n_tokens:
                break
        if m.generated < n_tokens:
            m.generated += 1     # resampled next_tok
            ctx_seed += 1
    return m


def baseline_decode(n_tokens: int, rng: random.Random,
                    target: TargetModel) -> Metrics:
    m = Metrics()
    ctx_seed = 1
    while m.generated < n_tokens:
        target.calls += 1
        m.target_calls += 1
        dist = target.distribution(ctx_seed)
        _ = sample(dist, rng)
        m.generated += 1
        ctx_seed += 1
    return m


# ---------------------------------------------------------------------------
# sweep  --  compare speedup across k and draft alignment
# ---------------------------------------------------------------------------

def main() -> None:
    n_tokens = 500
    print(f"=== decode {n_tokens} tokens, compare baseline vs speculative ===")

    target = TargetModel()
    rng = random.Random(7)
    base = baseline_decode(n_tokens, rng, target)
    print(f"baseline: {base.target_calls} target calls, "
          f"{base.tokens_per_target_call():.2f} tok/call")

    for alignment in (0.60, 0.75, 0.90):
        for k in (2, 4, 6):
            target = TargetModel()
            draft = DraftModel(alignment=alignment)
            rng = random.Random(7)
            m = speculative_decode(n_tokens, k, rng, target, draft)
            speedup = base.target_calls / max(1, m.target_calls)
            print(f"  align={alignment:.2f} k={k}  "
                  f"target_calls={m.target_calls:3d}  "
                  f"acceptance={m.acceptance_rate(k):.2f}  "
                  f"tok/call={m.tokens_per_target_call():.2f}  "
                  f"speedup={speedup:.2f}x")


if __name__ == "__main__":
    main()