Get Started

Serving Engine Internals, PagedAttention, Continuous Batching, Chunked Prefill

Modern serving-engine throughput rests on three compounding defaults, not a single trick. PagedAttention is always on. Continuous batching injects new requests into the active batch between decode iterations. Chunked prefill slices long prompts so decode tokens never starve. Turn all three on and a Llama 3.3 70B FP8 on one H100 SXM5 pushes 2,200-2,400 tok/s at 128 concurrent, roughly 25% above vLLM's own default and 3-4x a naive PyTorch loop. This lesson reads the scheduler and attention kernel of vLLM, the reference engine for all three techniques, at a level you can diagram, and ends with a toy continuous batcher in code/main.py that schedules prefill and decode the way vLLM does.

75 minutesLearnPython (stdlib, toy continuous batching scheduler)Free lesson

Learning Objectives

  • Explain PagedAttention as a KV cache allocator: blocks, block tables, and why fragmentation stays under 4% at production load.
  • Diagram continuous batching at the iteration level: how finished sequences leave the batch and new ones join without draining.
  • Describe chunked prefill in one sentence and name which latency metric it protects (hint: it is TTFT tail, not mean throughput).
  • Check a vLLM feature combination against the compatibility matrix for your version before enabling every optimization at once.

The Problem

A naive PyTorch serve loop runs one request at a time: tokenize, prefill, decode until EOS, return. At one user this works. At one hundred, it is a queue of patient people. The obvious fix, static batching, pads every request to the longest prompt in the window, pads every decode to the longest expected output, and stalls the whole batch on the slowest sequence. You pay for padding you never use, and fast requests wait for slow ones.

vLLM solves three problems at once. PagedAttention stops KV cache fragmentation from eating 60-80% of GPU memory the way classic contiguous allocation does. Continuous batching lets requests join and leave the batch between each decode iteration, so the batch is always full of real work. Chunked prefill breaks a 32k-token prompt into ~512-token slices that interleave with decode, so a long prompt does not freeze every decode token on the GPU.

The 2026 production default is all three on. You need to understand what each one does because the failure modes are all on the scheduler, not the model.

The Concept

PagedAttention as a virtual memory system

A KV cache is num_layers × 2 × num_heads × head_dim × seq_len × bytes_per_element per sequence. For Llama 3.3 70B at 8192 tokens, that is roughly 1.25 GB per sequence in BF16. If you pre-reserve 8192 slots for every request but the average request only uses 1500 tokens, you waste roughly 82% of the HBM you reserved. Classic batching pays this waste.

PagedAttention borrows the idea from OS virtual memory. KV cache is not contiguous per sequence. It is allocated in fixed-size blocks (default 16 tokens). Each sequence has a block table that maps its logical token positions to physical block IDs. When a sequence grows past its allocated blocks, one more block is added. When it finishes, its blocks return to the pool.

Fragmentation drops from 60-80% (classic) to under 4% (PagedAttention). You do not enable PagedAttention with a flag, it is the only allocator vLLM ships. The knob is --gpu-memory-utilization (default 0.9), which tells vLLM how much HBM to reserve for KV blocks after loading weights and activations.

Continuous batching at the iteration level

The old "dynamic batching" waited for a window (say 10 ms) to fill a batch, then ran prefill + decode + decode + decode until every sequence finished. Fast sequences left early and sat idle while the GPU finished the slow ones.

Continuous batching operates between each decode step. Call the set of running sequences the RUNNING list. At each iteration:

  1. Any sequence in RUNNING that just hit EOS or max_tokens is removed.
  2. The scheduler looks at the waiting queue. If there are free KV blocks, it admits new sequences (prefill or resumed).
  3. The forward pass runs on whatever is now in RUNNINGemitting one new token per sequence.

The batch size is never padded to a fixed number. Sequences at different positions in their output share one fused forward. In 2026 vLLM this is called the V1 scheduler. The key invariant: the scheduler runs once per decode iteration, not once per request.

Chunked prefill protects TTFT tail

Prefill is compute-bound. A 32k-token prompt on Llama 3.3 70B takes ~800 ms of pure prefill on one H100. While prefill runs, decode tokens for every other sequence in the batch wait. In a serving loop, the first-token latency (TTFT) of one long prompt becomes the inter-token latency (ITL) blip for dozens of other users.

Chunked prefill splits prefill into fixed-size chunks (default 512 tokens) and schedules each chunk as a unit. Between chunks the scheduler can advance decode sequences by one token. You trade a small absolute prefill latency hit (a few ms per chunk) for much lower decode-time jitter. P99 ITL under mixed load drops from ~50 ms to ~15 ms in published benchmarks.

The three defaults interact

All three features assume each other. PagedAttention gives the scheduler a fine-grained KV resource to trade against. Continuous batching needs that fine-grained resource so admitting a new sequence does not force a global reshuffle. Chunked prefill is a decision the scheduler makes on the same RUNNING list, it is one more scheduler policy, not a separate system.

You do not need to know every flag. You need to know what the scheduler optimizes: goodput under KV-block budget, subject to chunked prefill slicing.

Check the compatibility matrix

Check every feature combination against the compatibility matrix for your exact vLLM version before enabling all of them at once, because what composes changes between releases. In v0.18.0 the feature matrix marks speculative decoding as compatible with chunked prefill and prefix caching, and the speculative-decoding page lists two known incompatibilities: pipeline parallelism through v0.15.0, and draft-model speculation through v0.10.0. For the draft method itself, the 2026 default is often EAGLE-3 ("method": "eagle3"), covered in Phase 17 · 05.

Numbers you should remember

  • Llama 3.3 70B FP8, H100 SXM5, 128 concurrent, all three on: 2,200-2,400 tok/s.
  • Same model, default vLLM (no chunked prefill): ~1,800 tok/s.
  • Same model, naive PyTorch forward loop: ~600 tok/s.
  • KV fragmentation waste under PagedAttention at production load: <4%.
  • P99 ITL under mixed load: ~15 ms with chunked prefill, ~50 ms without.

What the scheduler looks like

Code
while True:
    finished = [s for s in RUNNING if s.is_done()]
    for s in finished: release_blocks(s); RUNNING.remove(s)

    while WAITING and have_free_blocks_for(WAITING[0]):
        s = WAITING.pop(0)
        allocate_initial_blocks(s)
        RUNNING.append(s)

    # schedule prefill chunks + decode in one batch
    batch = []
    for s in RUNNING:
        if s.in_prefill:
            batch.append(next_prefill_chunk(s))   # e.g. 512 tokens
        else:
            batch.append(decode_one_token(s))     # 1 token

    run_forward(batch)                            # one fused GPU call

code/main.py is exactly this loop in stdlib Python with fake token counts and fake forward latency. Running it shows how chunked prefill keeps decode sequences alive during a long prefill.

Use It

code/main.py simulates a vLLM-style scheduler with toggleable features. Run it to see:

  • NAIVE mode: one request at a time, no batching.
  • STATIC mode: pad and wait, classic batching.
  • CONTINUOUS mode: iteration-level admission and release.
  • CONTINUOUS + CHUNKED mode: prefill slices interleaved with decode.

The output shows total throughput (tokens per virtual second), TTFT mean, and P99 ITL. The CONTINUOUS + CHUNKED row should dominate on mixed traffic.

Ship It

This lesson produces outputs/skill-vllm-scheduler-reader.md. Given a serving config (batch size, KV memory utilization, chunked prefill size, speculative config), it produces a scheduler diagnosis that names which of the three defaults is bottlenecking and what to tune.

Exercises

  1. Run code/main.py. Compare STATIC to CONTINUOUS on a workload with mixed short and long requests. Where does the throughput gap come from, prefill efficiency, decode efficiency, or tail latency?
  2. Modify the toy scheduler to add --max-num-batched-tokens. What is the right value for an H100 running Llama 3.3 70B FP8? (Hint: it is a function of KV block size and number of free blocks, not raw HBM.)
  3. Re-read the vLLM v0.18.0 release notes. Which combinations of flags are mutually exclusive? List them.
  4. Compute the KV cache fragmentation waste for a trace of 1,000 requests with mean 1,500 output tokens, std 600 tokens, under (a) contiguous per-request allocation at 8192 max, (b) PagedAttention with 16-token blocks.
  5. Explain in one paragraph why chunked prefill helps P99 ITL but not throughput in isolation. Where does the throughput win come from in practice?

Key Terms

TermWhat people sayWhat it actually means
PagedAttention"the KV trick"Fixed-size block allocator for KV cache; fragmentation <4%
Block table"the page table"Per-sequence map from logical token position to physical KV block
Continuous batching"dynamic batching, but right"Admit/release decisions made every decode iteration
Chunked prefill"prefill splitting"Break long prefill into 512-token slices interleaved with decode
TTFT"first token time"Prefill + queue + network; dominated by prefill at long prompts
ITL"inter-token latency"Time between consecutive decode tokens; dominated by batch size
Goodput"throughput that meets SLO"Tokens/sec where every request still hit TTFT and ITL targets
V1 scheduler"the new scheduler"vLLM's 2026 scheduler; runs continuous batching with chunked prefill
--gpu-memory-utilization"the memory knob"Fraction of HBM reserved for KV blocks after weights and activations

Further Reading

Check what you learned

6 questions. Answer each one to see why.

Question 1 of 60 correct

What is the main problem classic static batching has with mixed-length requests?

The full code

Every file for this lesson. Copy it, run it, then change it.

main.py
"""Toy continuous-batching scheduler — stdlib Python.

Simulates four serving modes on the same workload:
  NAIVE            : one request at a time, no batching
  STATIC           : pad to batch boundary, wait for slowest
  CONTINUOUS       : iteration-level admit/release
  CONTINUOUS+CHUNK : continuous + chunked prefill (512-token slices)

Reports throughput (tok / virt-sec), mean TTFT, and P99 ITL so you can
reproduce the shape of the vLLM benchmarks without a GPU. Pedagogical:
the latency constants are illustrative, not measured.
"""

from __future__ import annotations

from dataclasses import dataclass, field
from collections import deque
import random
import statistics


FORWARD_LATENCY_PER_TOKEN = 0.0005   # 0.5 ms per decode token in the batch
PREFILL_LATENCY_PER_TOKEN = 0.00004  # prefill ~12x cheaper per token than decode
BATCH_OVERHEAD = 0.0002              # fixed overhead per forward call
CHUNK_SIZE = 512
KV_BLOCK_SIZE = 16
KV_BLOCKS_AVAILABLE = 1800           # toy KV block budget


@dataclass
class Request:
    req_id: int
    prompt_len: int
    output_len: int
    arrived_at: float
    prefilled: int = 0
    generated: int = 0
    ttft: float | None = None
    last_token_at: float | None = None
    itl_samples: list[float] = field(default_factory=list)

    @property
    def in_prefill(self) -> bool:
        return self.prefilled < self.prompt_len

    @property
    def done(self) -> bool:
        return self.generated >= self.output_len

    def blocks_needed(self) -> int:
        total = self.prompt_len + self.output_len
        return (total + KV_BLOCK_SIZE - 1) // KV_BLOCK_SIZE


def make_workload(n: int = 60, seed: int = 7) -> list[Request]:
    rng = random.Random(seed)
    reqs = []
    now = 0.0
    for i in range(n):
        now += rng.expovariate(40.0)   # ~40 req/s arrival
        prompt_len = rng.choice([128, 256, 512, 2048, 8192])
        out_len = rng.randint(50, 300)
        reqs.append(Request(i, prompt_len, out_len, now))
    return reqs


def report(label: str, reqs: list[Request], sim_end: float) -> None:
    ttfts = [r.ttft - r.arrived_at for r in reqs if r.ttft is not None]
    itls = [dt for r in reqs for dt in r.itl_samples]
    total_out = sum(r.generated for r in reqs)
    throughput = total_out / sim_end if sim_end else 0
    mean_ttft = statistics.mean(ttfts) * 1000 if ttfts else 0
    p99_itl = sorted(itls)[int(0.99 * len(itls)) - 1] * 1000 if itls else 0
    print(f"{label:28}  throughput={throughput:6.0f} tok/s   "
          f"mean_TTFT={mean_ttft:6.1f} ms   "
          f"P99_ITL={p99_itl:5.1f} ms   finished={sum(r.done for r in reqs)}/{len(reqs)}")


def simulate_naive(reqs: list[Request]) -> float:
    """One at a time. Prefill the whole prompt, then decode until done."""
    now = 0.0
    for r in reqs:
        if now < r.arrived_at:
            now = r.arrived_at
        now += r.prompt_len * PREFILL_LATENCY_PER_TOKEN + BATCH_OVERHEAD
        r.prefilled = r.prompt_len
        r.ttft = now
        r.last_token_at = now
        for _ in range(r.output_len):
            prev = r.last_token_at
            now += FORWARD_LATENCY_PER_TOKEN + BATCH_OVERHEAD
            r.generated += 1
            r.itl_samples.append(now - prev)
            r.last_token_at = now
    return now


def simulate_static(reqs: list[Request], batch: int = 16) -> float:
    """Group into fixed batches; wait for the slowest to finish."""
    now = 0.0
    i = 0
    while i < len(reqs):
        window = reqs[i:i + batch]
        i += batch
        now = max(now, max(r.arrived_at for r in window))
        pad_prompt = max(r.prompt_len for r in window)
        pad_output = max(r.output_len for r in window)
        now += pad_prompt * PREFILL_LATENCY_PER_TOKEN + BATCH_OVERHEAD
        for r in window:
            r.prefilled = r.prompt_len
            r.ttft = now
            r.last_token_at = now
        for _ in range(pad_output):
            prev_now = now
            now += FORWARD_LATENCY_PER_TOKEN * len(window) / 16 + BATCH_OVERHEAD
            for r in window:
                if r.generated < r.output_len:
                    r.generated += 1
                    r.itl_samples.append(now - prev_now)
                    r.last_token_at = now
    return now


def simulate_continuous(reqs: list[Request], chunked: bool) -> float:
    waiting = deque(sorted(reqs, key=lambda r: r.arrived_at))
    running: list[Request] = []
    blocks_used = 0
    now = 0.0
    while waiting or running:
        if waiting and running and now < waiting[0].arrived_at and not running:
            now = waiting[0].arrived_at
        while waiting and waiting[0].arrived_at <= now:
            r = waiting[0]
            if blocks_used + r.blocks_needed() > KV_BLOCKS_AVAILABLE:
                break
            blocks_used += r.blocks_needed()
            running.append(waiting.popleft())
        if not running:
            if not waiting:
                break
            now = waiting[0].arrived_at
            continue

        batch_tokens = 0
        prefill_work = 0
        decoded: list[Request] = []
        for r in running:
            if r.in_prefill:
                remaining = r.prompt_len - r.prefilled
                take = min(CHUNK_SIZE if chunked else remaining, remaining)
                r.prefilled += take
                prefill_work += take
                if r.prefilled >= r.prompt_len:
                    r.ttft = now + prefill_work * PREFILL_LATENCY_PER_TOKEN
            else:
                decoded.append(r)
                batch_tokens += 1

        dt = (prefill_work * PREFILL_LATENCY_PER_TOKEN
              + batch_tokens * FORWARD_LATENCY_PER_TOKEN
              + BATCH_OVERHEAD)
        now += dt
        for r in decoded:
            prev = r.last_token_at or r.ttft or now
            r.generated += 1
            r.itl_samples.append(now - prev)
            r.last_token_at = now
            if r.ttft is None:
                r.ttft = now

        finished = [r for r in running if r.done]
        for r in finished:
            blocks_used -= r.blocks_needed()
            running.remove(r)
    return now


def main() -> None:
    print("=" * 80)
    print("TOY vLLM SCHEDULER — four modes on the same 60-request workload")
    print("=" * 80)

    base = make_workload()
    w1 = [Request(r.req_id, r.prompt_len, r.output_len, r.arrived_at) for r in base]
    end = simulate_naive(w1)
    report("NAIVE", w1, end)

    w2 = [Request(r.req_id, r.prompt_len, r.output_len, r.arrived_at) for r in base]
    end = simulate_static(w2)
    report("STATIC (batch=16, padded)", w2, end)

    w3 = [Request(r.req_id, r.prompt_len, r.output_len, r.arrived_at) for r in base]
    end = simulate_continuous(w3, chunked=False)
    report("CONTINUOUS (no chunk)", w3, end)

    w4 = [Request(r.req_id, r.prompt_len, r.output_len, r.arrived_at) for r in base]
    end = simulate_continuous(w4, chunked=True)
    report("CONTINUOUS + CHUNKED", w4, end)

    print()
    print("Read the CONTINUOUS+CHUNKED row. That is what vLLM ships as default.")
    print("The gap between STATIC and CONTINUOUS is the whole reason vLLM exists.")


if __name__ == "__main__":
    main()