Serving Engine Internals, PagedAttention, Continuous Batching, Chunked Prefill
Modern serving-engine throughput rests on three compounding defaults, not a single trick. PagedAttention is always on. Continuous batching injects new requests into the active batch between decode iterations. Chunked prefill slices long prompts so decode tokens never starve. Turn all three on and a Llama 3.3 70B FP8 on one H100 SXM5 pushes 2,200-2,400 tok/s at 128 concurrent, roughly 25% above vLLM's own default and 3-4x a naive PyTorch loop. This lesson reads the scheduler and attention kernel of vLLM, the reference engine for all three techniques, at a level you can diagram, and ends with a toy continuous batcher in code/main.py that schedules prefill and decode the way vLLM does.
Learning Objectives
- Explain PagedAttention as a KV cache allocator: blocks, block tables, and why fragmentation stays under 4% at production load.
- Diagram continuous batching at the iteration level: how finished sequences leave the batch and new ones join without draining.
- Describe chunked prefill in one sentence and name which latency metric it protects (hint: it is TTFT tail, not mean throughput).
- Check a vLLM feature combination against the compatibility matrix for your version before enabling every optimization at once.
The Problem
A naive PyTorch serve loop runs one request at a time: tokenize, prefill, decode until EOS, return. At one user this works. At one hundred, it is a queue of patient people. The obvious fix, static batching, pads every request to the longest prompt in the window, pads every decode to the longest expected output, and stalls the whole batch on the slowest sequence. You pay for padding you never use, and fast requests wait for slow ones.
vLLM solves three problems at once. PagedAttention stops KV cache fragmentation from eating 60-80% of GPU memory the way classic contiguous allocation does. Continuous batching lets requests join and leave the batch between each decode iteration, so the batch is always full of real work. Chunked prefill breaks a 32k-token prompt into ~512-token slices that interleave with decode, so a long prompt does not freeze every decode token on the GPU.
The 2026 production default is all three on. You need to understand what each one does because the failure modes are all on the scheduler, not the model.
The Concept
PagedAttention as a virtual memory system
A KV cache is num_layers × 2 × num_heads × head_dim × seq_len × bytes_per_element per sequence. For Llama 3.3 70B at 8192 tokens, that is roughly 1.25 GB per sequence in BF16. If you pre-reserve 8192 slots for every request but the average request only uses 1500 tokens, you waste roughly 82% of the HBM you reserved. Classic batching pays this waste.
PagedAttention borrows the idea from OS virtual memory. KV cache is not contiguous per sequence. It is allocated in fixed-size blocks (default 16 tokens). Each sequence has a block table that maps its logical token positions to physical block IDs. When a sequence grows past its allocated blocks, one more block is added. When it finishes, its blocks return to the pool.
Fragmentation drops from 60-80% (classic) to under 4% (PagedAttention). You do not enable PagedAttention with a flag, it is the only allocator vLLM ships. The knob is --gpu-memory-utilization (default 0.9), which tells vLLM how much HBM to reserve for KV blocks after loading weights and activations.
Continuous batching at the iteration level
The old "dynamic batching" waited for a window (say 10 ms) to fill a batch, then ran prefill + decode + decode + decode until every sequence finished. Fast sequences left early and sat idle while the GPU finished the slow ones.
Continuous batching operates between each decode step. Call the set of running sequences the RUNNING list. At each iteration:
- Any sequence in
RUNNINGthat just hit EOS or max_tokens is removed. - The scheduler looks at the waiting queue. If there are free KV blocks, it admits new sequences (prefill or resumed).
- The forward pass runs on whatever is now in
RUNNINGemitting one new token per sequence.
The batch size is never padded to a fixed number. Sequences at different positions in their output share one fused forward. In 2026 vLLM this is called the V1 scheduler. The key invariant: the scheduler runs once per decode iteration, not once per request.
Chunked prefill protects TTFT tail
Prefill is compute-bound. A 32k-token prompt on Llama 3.3 70B takes ~800 ms of pure prefill on one H100. While prefill runs, decode tokens for every other sequence in the batch wait. In a serving loop, the first-token latency (TTFT) of one long prompt becomes the inter-token latency (ITL) blip for dozens of other users.
Chunked prefill splits prefill into fixed-size chunks (default 512 tokens) and schedules each chunk as a unit. Between chunks the scheduler can advance decode sequences by one token. You trade a small absolute prefill latency hit (a few ms per chunk) for much lower decode-time jitter. P99 ITL under mixed load drops from ~50 ms to ~15 ms in published benchmarks.
The three defaults interact
All three features assume each other. PagedAttention gives the scheduler a fine-grained KV resource to trade against. Continuous batching needs that fine-grained resource so admitting a new sequence does not force a global reshuffle. Chunked prefill is a decision the scheduler makes on the same RUNNING list, it is one more scheduler policy, not a separate system.
You do not need to know every flag. You need to know what the scheduler optimizes: goodput under KV-block budget, subject to chunked prefill slicing.
Check the compatibility matrix
Check every feature combination against the compatibility matrix for your exact vLLM version before enabling all of them at once, because what composes changes between releases. In v0.18.0 the feature matrix marks speculative decoding as compatible with chunked prefill and prefix caching, and the speculative-decoding page lists two known incompatibilities: pipeline parallelism through v0.15.0, and draft-model speculation through v0.10.0. For the draft method itself, the 2026 default is often EAGLE-3 ("method": "eagle3"), covered in Phase 17 · 05.
Numbers you should remember
- Llama 3.3 70B FP8, H100 SXM5, 128 concurrent, all three on: 2,200-2,400 tok/s.
- Same model, default vLLM (no chunked prefill): ~1,800 tok/s.
- Same model, naive PyTorch forward loop: ~600 tok/s.
- KV fragmentation waste under PagedAttention at production load: <4%.
- P99 ITL under mixed load: ~15 ms with chunked prefill, ~50 ms without.
What the scheduler looks like
while True:
finished = [s for s in RUNNING if s.is_done()]
for s in finished: release_blocks(s); RUNNING.remove(s)
while WAITING and have_free_blocks_for(WAITING[0]):
s = WAITING.pop(0)
allocate_initial_blocks(s)
RUNNING.append(s)
# schedule prefill chunks + decode in one batch
batch = []
for s in RUNNING:
if s.in_prefill:
batch.append(next_prefill_chunk(s)) # e.g. 512 tokens
else:
batch.append(decode_one_token(s)) # 1 token
run_forward(batch) # one fused GPU callcode/main.py is exactly this loop in stdlib Python with fake token counts and fake forward latency. Running it shows how chunked prefill keeps decode sequences alive during a long prefill.
Use It
code/main.py simulates a vLLM-style scheduler with toggleable features. Run it to see:
NAIVEmode: one request at a time, no batching.STATICmode: pad and wait, classic batching.CONTINUOUSmode: iteration-level admission and release.CONTINUOUS + CHUNKEDmode: prefill slices interleaved with decode.
The output shows total throughput (tokens per virtual second), TTFT mean, and P99 ITL. The CONTINUOUS + CHUNKED row should dominate on mixed traffic.
Ship It
This lesson produces outputs/skill-vllm-scheduler-reader.md. Given a serving config (batch size, KV memory utilization, chunked prefill size, speculative config), it produces a scheduler diagnosis that names which of the three defaults is bottlenecking and what to tune.
Exercises
- Run
code/main.py. CompareSTATICtoCONTINUOUSon a workload with mixed short and long requests. Where does the throughput gap come from, prefill efficiency, decode efficiency, or tail latency? - Modify the toy scheduler to add
--max-num-batched-tokens. What is the right value for an H100 running Llama 3.3 70B FP8? (Hint: it is a function of KV block size and number of free blocks, not raw HBM.) - Re-read the vLLM v0.18.0 release notes. Which combinations of flags are mutually exclusive? List them.
- Compute the KV cache fragmentation waste for a trace of 1,000 requests with mean 1,500 output tokens, std 600 tokens, under (a) contiguous per-request allocation at 8192 max, (b) PagedAttention with 16-token blocks.
- Explain in one paragraph why chunked prefill helps P99 ITL but not throughput in isolation. Where does the throughput win come from in practice?
Key Terms
| Term | What people say | What it actually means |
|---|---|---|
| PagedAttention | "the KV trick" | Fixed-size block allocator for KV cache; fragmentation <4% |
| Block table | "the page table" | Per-sequence map from logical token position to physical KV block |
| Continuous batching | "dynamic batching, but right" | Admit/release decisions made every decode iteration |
| Chunked prefill | "prefill splitting" | Break long prefill into 512-token slices interleaved with decode |
| TTFT | "first token time" | Prefill + queue + network; dominated by prefill at long prompts |
| ITL | "inter-token latency" | Time between consecutive decode tokens; dominated by batch size |
| Goodput | "throughput that meets SLO" | Tokens/sec where every request still hit TTFT and ITL targets |
| V1 scheduler | "the new scheduler" | vLLM's 2026 scheduler; runs continuous batching with chunked prefill |
--gpu-memory-utilization | "the memory knob" | Fraction of HBM reserved for KV blocks after weights and activations |
Further Reading
- vLLM documentation, Speculative Decodingofficial source on chunked-prefill and speculative-decoding compatibility.
- vLLM Release Notes (NVIDIA)2026 release cadence and version-specific behavior.
- vLLM Blog, PagedAttentionthe original write-up that still defines how to think about the allocator.
- PagedAttention paper (arXiv:2309.06180)fragmentation analysis and scheduler design.
- Aleksa Gordic, Inside vLLMdetailed V1 scheduler walkthrough with flame graphs.
Check what you learned
6 questions. Answer each one to see why.
What is the main problem classic static batching has with mixed-length requests?
The full code
Every file for this lesson. Copy it, run it, then change it.
"""Toy continuous-batching scheduler — stdlib Python.
Simulates four serving modes on the same workload:
NAIVE : one request at a time, no batching
STATIC : pad to batch boundary, wait for slowest
CONTINUOUS : iteration-level admit/release
CONTINUOUS+CHUNK : continuous + chunked prefill (512-token slices)
Reports throughput (tok / virt-sec), mean TTFT, and P99 ITL so you can
reproduce the shape of the vLLM benchmarks without a GPU. Pedagogical:
the latency constants are illustrative, not measured.
"""
from __future__ import annotations
from dataclasses import dataclass, field
from collections import deque
import random
import statistics
FORWARD_LATENCY_PER_TOKEN = 0.0005 # 0.5 ms per decode token in the batch
PREFILL_LATENCY_PER_TOKEN = 0.00004 # prefill ~12x cheaper per token than decode
BATCH_OVERHEAD = 0.0002 # fixed overhead per forward call
CHUNK_SIZE = 512
KV_BLOCK_SIZE = 16
KV_BLOCKS_AVAILABLE = 1800 # toy KV block budget
@dataclass
class Request:
req_id: int
prompt_len: int
output_len: int
arrived_at: float
prefilled: int = 0
generated: int = 0
ttft: float | None = None
last_token_at: float | None = None
itl_samples: list[float] = field(default_factory=list)
@property
def in_prefill(self) -> bool:
return self.prefilled < self.prompt_len
@property
def done(self) -> bool:
return self.generated >= self.output_len
def blocks_needed(self) -> int:
total = self.prompt_len + self.output_len
return (total + KV_BLOCK_SIZE - 1) // KV_BLOCK_SIZE
def make_workload(n: int = 60, seed: int = 7) -> list[Request]:
rng = random.Random(seed)
reqs = []
now = 0.0
for i in range(n):
now += rng.expovariate(40.0) # ~40 req/s arrival
prompt_len = rng.choice([128, 256, 512, 2048, 8192])
out_len = rng.randint(50, 300)
reqs.append(Request(i, prompt_len, out_len, now))
return reqs
def report(label: str, reqs: list[Request], sim_end: float) -> None:
ttfts = [r.ttft - r.arrived_at for r in reqs if r.ttft is not None]
itls = [dt for r in reqs for dt in r.itl_samples]
total_out = sum(r.generated for r in reqs)
throughput = total_out / sim_end if sim_end else 0
mean_ttft = statistics.mean(ttfts) * 1000 if ttfts else 0
p99_itl = sorted(itls)[int(0.99 * len(itls)) - 1] * 1000 if itls else 0
print(f"{label:28} throughput={throughput:6.0f} tok/s "
f"mean_TTFT={mean_ttft:6.1f} ms "
f"P99_ITL={p99_itl:5.1f} ms finished={sum(r.done for r in reqs)}/{len(reqs)}")
def simulate_naive(reqs: list[Request]) -> float:
"""One at a time. Prefill the whole prompt, then decode until done."""
now = 0.0
for r in reqs:
if now < r.arrived_at:
now = r.arrived_at
now += r.prompt_len * PREFILL_LATENCY_PER_TOKEN + BATCH_OVERHEAD
r.prefilled = r.prompt_len
r.ttft = now
r.last_token_at = now
for _ in range(r.output_len):
prev = r.last_token_at
now += FORWARD_LATENCY_PER_TOKEN + BATCH_OVERHEAD
r.generated += 1
r.itl_samples.append(now - prev)
r.last_token_at = now
return now
def simulate_static(reqs: list[Request], batch: int = 16) -> float:
"""Group into fixed batches; wait for the slowest to finish."""
now = 0.0
i = 0
while i < len(reqs):
window = reqs[i:i + batch]
i += batch
now = max(now, max(r.arrived_at for r in window))
pad_prompt = max(r.prompt_len for r in window)
pad_output = max(r.output_len for r in window)
now += pad_prompt * PREFILL_LATENCY_PER_TOKEN + BATCH_OVERHEAD
for r in window:
r.prefilled = r.prompt_len
r.ttft = now
r.last_token_at = now
for _ in range(pad_output):
prev_now = now
now += FORWARD_LATENCY_PER_TOKEN * len(window) / 16 + BATCH_OVERHEAD
for r in window:
if r.generated < r.output_len:
r.generated += 1
r.itl_samples.append(now - prev_now)
r.last_token_at = now
return now
def simulate_continuous(reqs: list[Request], chunked: bool) -> float:
waiting = deque(sorted(reqs, key=lambda r: r.arrived_at))
running: list[Request] = []
blocks_used = 0
now = 0.0
while waiting or running:
if waiting and running and now < waiting[0].arrived_at and not running:
now = waiting[0].arrived_at
while waiting and waiting[0].arrived_at <= now:
r = waiting[0]
if blocks_used + r.blocks_needed() > KV_BLOCKS_AVAILABLE:
break
blocks_used += r.blocks_needed()
running.append(waiting.popleft())
if not running:
if not waiting:
break
now = waiting[0].arrived_at
continue
batch_tokens = 0
prefill_work = 0
decoded: list[Request] = []
for r in running:
if r.in_prefill:
remaining = r.prompt_len - r.prefilled
take = min(CHUNK_SIZE if chunked else remaining, remaining)
r.prefilled += take
prefill_work += take
if r.prefilled >= r.prompt_len:
r.ttft = now + prefill_work * PREFILL_LATENCY_PER_TOKEN
else:
decoded.append(r)
batch_tokens += 1
dt = (prefill_work * PREFILL_LATENCY_PER_TOKEN
+ batch_tokens * FORWARD_LATENCY_PER_TOKEN
+ BATCH_OVERHEAD)
now += dt
for r in decoded:
prev = r.last_token_at or r.ttft or now
r.generated += 1
r.itl_samples.append(now - prev)
r.last_token_at = now
if r.ttft is None:
r.ttft = now
finished = [r for r in running if r.done]
for r in finished:
blocks_used -= r.blocks_needed()
running.remove(r)
return now
def main() -> None:
print("=" * 80)
print("TOY vLLM SCHEDULER — four modes on the same 60-request workload")
print("=" * 80)
base = make_workload()
w1 = [Request(r.req_id, r.prompt_len, r.output_len, r.arrived_at) for r in base]
end = simulate_naive(w1)
report("NAIVE", w1, end)
w2 = [Request(r.req_id, r.prompt_len, r.output_len, r.arrived_at) for r in base]
end = simulate_static(w2)
report("STATIC (batch=16, padded)", w2, end)
w3 = [Request(r.req_id, r.prompt_len, r.output_len, r.arrived_at) for r in base]
end = simulate_continuous(w3, chunked=False)
report("CONTINUOUS (no chunk)", w3, end)
w4 = [Request(r.req_id, r.prompt_len, r.output_len, r.arrived_at) for r in base]
end = simulate_continuous(w4, chunked=True)
report("CONTINUOUS + CHUNKED", w4, end)
print()
print("Read the CONTINUOUS+CHUNKED row. That is what vLLM ships as default.")
print("The gap between STATIC and CONTINUOUS is the whole reason vLLM exists.")
if __name__ == "__main__":
main()