Get Started

Hardware-Specialized Inference Compilation, FP8 and NVFP4 on Blackwell

Hardware-specialized inference compilation trades portability for throughput, and TensorRT-LLM, NVIDIA-only, tuned for Blackwell, is the clearest example of the trade paying off. On GB200 NVL72 with Dynamo orchestration, SemiAnalysis InferenceX measured $0.012 per million tokens on a 120B model in Q1-Q2 2026, against $0.09/M on H100 + vLLM, a 7x economic gap. The stack is three floating-point regimes compounded: FP8 stays critical for KV cache and attention kernels because it has the dynamic range they need; NVFP4 (4-bit microscaling) handles weights and activations; multi-token prediction (MTP) and disaggregated prefill/decode add another 2-3x on top. Day-0 model support loads FP4 weights directly without post-training conversion. The catch for 2026 engineering teams: TRT-LLM is open-source but NVIDIA-specific, CUDA- and Blackwell-specialized, so adopting it trades portability for throughput. Run the math on your mix of models and hardware before committing.

75 minutesLearnPython (stdlib, toy FP8/NVFP4 memory and cost calculator)Free lesson

Learning Objectives

  • Explain why FP8 stays critical for KV cache and attention even when weights are in NVFP4.
  • Compute the HBM footprint of a frontier model under BF16, FP8, and NVFP4 and reason about where the savings come from.
  • Name the Blackwell-specific features TRT-LLM exploits (day-0 FP4, MTP, disaggregated serving, all-to-all primitives).
  • Decide when TRT-LLM's NVIDIA-lock is worth the 7x cost gap vs vLLM on Hopper.

The Problem

The frontier of inference economics in 2026 is "how many tokens per dollar". The answer depends on four stacked choices: hardware generation (Hopper H100/H200 vs Blackwell B200/GB200), precision (BF16 → FP8 → NVFP4), serving engine (vLLM vs SGLang vs TRT-LLM), and orchestration (plain vs disaggregated vs Dynamo).

On Hopper with vLLM, a 120B MoE runs at ~$0.09 per million tokens. On Blackwell with TRT-LLM + Dynamo, the same model runs at ~$0.012 to 7x cheaper. Some of that gap is hardware (Blackwell is 11-15x per-GPU LLM throughput vs Hopper). Some is the stack: FP4 weights, MTP draft, disaggregated prefill/decode, and NVLink 5 all-to-all for MoE expert communication.

You cannot replicate this outside NVIDIA's stack. That is the tradeoff, portability for economics. Understanding which stack choices give which share of the gap is the point of this lesson.

The Concept

Why FP8 is still the floor for KV cache

A common mistake in 2026: assuming NVFP4 applies everywhere. It does not. KV cache needs FP8 (8-bit floating point) because it stores attention keys and values that span a wide dynamic range. Quantizing KV to FP4 causes catastrophic accuracy loss, the tail of the distribution drops off and attention scores collapse. FP8's exponent bits give KV cache the range it needs.

NVFP4 (2025-2026) applies to weights and activations. Microscaling: each block of weights has its own scale factor so small blocks can span different dynamic ranges without per-tensor scale loss. For activations, FP4 holds up because activations are small-range within a layer.

The typical Blackwell config:

  • Weights: NVFP4 (4-bit microscaling).
  • Activations: NVFP4.
  • KV cache: FP8.
  • Attention accumulator: FP32 (softmax stability).

The Blackwell-specific primitives TRT-LLM uses

  • Day-0 FP4 weights: model providers ship FP4 weights directly; TRT-LLM loads without post-training conversion. No AWQ / GPTQ step for FP4.
  • Multi-token prediction (MTP): same idea as EAGLE (Phase 17 · 05) but integrated into the TRT-LLM build.
  • Disaggregated serving: prefill and decode on separate GPU pools, KV cache transferred over NVLink or InfiniBand. Same idea as Dynamo (Phase 17 · 20).
  • All-to-all communication primitives: NVLink 5 cut MoE expert communication latency by 3x vs Hopper. TRT-LLM's MoE kernels are tuned for this.
  • NVFP4 + MXFP8 microscaling: hardware-accelerated scale-factor handling on Blackwell Tensor Cores.

The numbers you should memorize

  • HGX B200 at $0.02/M tokens on GPT-OSS-120B via TRT-LLM.
  • GB200 NVL72 at $0.012/M tokens via Dynamo (orchestrating TRT-LLM).
  • H100 + vLLM ≈ $0.09/M tokens on comparable workload.
  • 2.8x throughput gain in three months of TRT-LLM updates (2026).
  • 11-15x per-GPU LLM throughput, Blackwell vs Hopper.
  • MLPerf Inference v6.0 (April 2026): Blackwell dominates every submitted task.

What FP4 actually costs in quality

NVFP4 is aggressive. On reasoning-heavy workloads (chain-of-thought, math, code-gen with long context), FP4 weights degrade visibly. Per-block calibration mitigates but does not eliminate. Teams shipping reasoning models often use FP8 weights + FP4 activations as a compromise, or stick to H200 with FP8 throughout.

The rule: always validate task quality on your eval set before committing to NVFP4 weights.

Why this is an NVIDIA-lock decision

TRT-LLM is C++ + CUDA + closed-source kernels. Models need to be compiled for a specific GPU SKU. No AMD, no Intel, no ARM. If your infra strategy is multi-vendor, TRT-LLM is a non-starter for the TRT-LLM-served tier, you can still serve from vLLM on mixed hardware. If you are NVIDIA-only, the 7x gap pays for the lock.

2026 practical recipe

For a $100M+ annual inference bill, running on Hopper + vLLM leaves 7-10x on the table. Migrate cost-dominant workloads to Blackwell + TRT-LLM + Dynamo. Keep experimentation tier on H100 + vLLM for model iteration speed. Validate quality on each NVFP4-converted model before production.

The disaggregation bonus

TRT-LLM's disaggregated serving (separate prefill and decode pools) is covered in depth in Phase 17 · 20. On Blackwell, the multiplier stacks: FP4 weights × MTP speedup × disaggregated placement × cache-aware routing. The 7x number assumes this full stack.

Use It

code/main.py computes HBM footprint, decode throughput (memory-bound regime), and $/M-tokens for a model across three stacks: H100 + BF16 + vLLM, H100 + FP8 + vLLM, B200 + NVFP4/FP8 + TRT-LLM. Run it to see the compounding effect and the share of the gap each change contributes.

Ship It

This lesson produces outputs/skill-trtllm-blackwell-advisor.md. Given a workload, model size, and annual token volume, it decides whether the Blackwell + TRT-LLM stack is worth the NVIDIA-lock.

Exercises

  1. Run code/main.py. On a 120B MoE with 30% active parameters, compute the memory-bandwidth-limited decode throughput on H100 BF16, H100 FP8, and B200 NVFP4/FP8. Where does the biggest jump come from?
  2. A customer spends $2M/year on H100 + vLLM. What is the break-even number of Blackwell GPUs they need to buy to amortize a migration to TRT-LLM in 12 months, given the 7x economic gap?
  3. You see accuracy drop 3 points on MATH after NVFP4 weight conversion. Name two recovery paths: one quality-first (keep FP8 weights), one cost-first (calibrate with in-domain data).
  4. Read the MLPerf v6.0 inference results. Which task has the smallest Blackwell-over-Hopper gap, and why?
  5. Compute the HBM needed for a 405B model at NVFP4 weights + FP8 KV cache at 128k context. Does it fit on a single GB200 NVL72 node?

Key Terms

TermWhat people sayWhat it actually means
FP8"eight-bit float"8-bit floating point; used for KV cache and attention due to dynamic range
NVFP4"four-bit micro"NVIDIA's 4-bit microscaling FP format; weights and activations on Blackwell
MXFP8"MX eight"Microscaling FP8 variant; hardware-accelerated on Blackwell Tensor Cores
Day-0 FP4"ship FP4 weights"Model providers release weights already in FP4; no post-train conversion step
MTP"multi-token prediction"TRT-LLM's integrated speculative-decoding draft (Phase 17 · 05)
Disaggregated serving"split prefill/decode"Prefill and decode on separate GPU pools; KV transferred over NVLink/IB
All-to-all"MoE expert comm"Communication pattern routing tokens to expert GPUs; NVLink 5 cuts 3x
InferenceX"SemiAnalysis inference bench"The 2026 industry-accepted cost-per-token benchmark

Further Reading

Check what you learned

6 questions. Answer each one to see why.

Question 1 of 60 correct

Roughly what is the per-million-tokens cost gap the lesson reports between Blackwell + TRT-LLM + Dynamo and H100 + vLLM on a comparable 120B-class workload?

The full code

Every file for this lesson. Copy it, run it, then change it.

main.py
"""Toy Blackwell + TRT-LLM economics calculator — stdlib Python.

Computes HBM footprint and decode throughput for a model under three stacks:
  H100 + BF16 + vLLM
  H100 + FP8 + vLLM
  B200 + NVFP4 weights / FP8 KV + TRT-LLM + Dynamo
  GB200 NVL72 + NVFP4 / FP8 + TRT-LLM + Dynamo

The decode-throughput model is memory-bandwidth-limited: tokens/sec is
proportional to HBM-bandwidth / bytes-per-token. Numbers are pedagogical
illustrations of the shape of the 2026 Blackwell economics.
"""

from __future__ import annotations

from dataclasses import dataclass


@dataclass
class Stack:
    name: str
    hbm_gb: int               # per-GPU HBM
    hbm_bw_tbs: float         # HBM bandwidth in TB/s
    weight_bits: float        # effective weight precision
    kv_bits: float            # KV cache precision
    mtp_factor: float         # 1.0 = no draft, 1.8 = MTP on
    disagg_factor: float      # additional throughput from disaggregation
    price_per_gpu_hour: float


STACKS = [
    Stack("H100 + BF16 + vLLM",           80, 3.35,  16, 16, 1.0,  1.0,  2.50),
    Stack("H100 + FP8 + vLLM",            80, 3.35,   8,  8, 1.0,  1.0,  2.50),
    Stack("H200 + FP8 + vLLM",           141, 4.80,   8,  8, 1.0,  1.0,  3.50),
    Stack("B200 + NVFP4 + FP8 + TRT-LLM", 192, 8.00,   4,  8, 1.8,  1.6,  4.80),
    Stack("GB200 NVL72 + TRT-LLM + Dyn", 192, 8.00,   4,  8, 1.8,  2.5,  6.20),
]


def hbm_footprint_gb(params_b: float, active_b: float, seq_len: int, stack: Stack) -> tuple[float, float]:
    weight_gb = params_b * stack.weight_bits / 8
    # KV cache for a typical head config: num_layers * 2 * num_kv_heads * head_dim * seq_len * bytes/element
    # Use a representative 70B shape scaled by active param size
    layers = 64 * (active_b / 35.0)**0.5
    kv_heads = 8
    head_dim = 128
    kv_gb = layers * 2 * kv_heads * head_dim * seq_len * (stack.kv_bits / 8) / 1e9
    return weight_gb, kv_gb


def decode_throughput(active_b: float, stack: Stack) -> float:
    """Tokens per second per GPU, memory-bandwidth-limited.
    Each decoded token reads `active_b * weight_bits/8` bytes of weights.
    """
    bytes_per_token = active_b * 1e9 * stack.weight_bits / 8
    raw_tokens_per_s = stack.hbm_bw_tbs * 1e12 / bytes_per_token
    return raw_tokens_per_s * stack.mtp_factor * stack.disagg_factor


def cost_per_million_tokens(active_b: float, stack: Stack) -> float:
    tps = decode_throughput(active_b, stack)
    tokens_per_hour = tps * 3600
    return stack.price_per_gpu_hour / tokens_per_hour * 1e6


def print_stack(params_b: float, active_b: float, seq_len: int = 8192) -> None:
    print(f"Model: {params_b}B total, {active_b}B active, {seq_len:,} tokens context")
    print("-" * 90)
    print(f"{'stack':40} {'W GB':>7} {'KV GB':>7} {'tok/s':>9} {'$/M tok':>10}")
    for s in STACKS:
        w, kv = hbm_footprint_gb(params_b, active_b, seq_len, s)
        tps = decode_throughput(active_b, s)
        cost = cost_per_million_tokens(active_b, s)
        fits = "" if (w + kv) <= s.hbm_gb else "  (multi-GPU)"
        print(f"{s.name:40} {w:7.1f} {kv:7.2f} {tps:9.0f} {cost:10.4f}{fits}")
    print()


def main() -> None:
    print("=" * 90)
    print("TOY BLACKWELL + TRT-LLM ECONOMICS — memory-bandwidth-limited decode")
    print("=" * 90)
    print()

    print_stack(70, 70)    # dense 70B
    print_stack(120, 36)   # GPT-OSS-120B MoE (30% active)
    print_stack(405, 405)  # Llama 3.1 405B dense
    print_stack(671, 37)   # DeepSeek-V3 scale MoE

    print("=" * 90)
    print("KEY FINDING")
    print("-" * 90)
    print("  The 7x cost gap stacks from four sources:")
    print("    1. HBM bandwidth (H100 3.35 TB/s vs B200 8.0 TB/s) ~2.4x")
    print("    2. NVFP4 weights (half the bytes per token)       ~2.0x")
    print("    3. MTP draft (~1.8x on accepted tokens)           ~1.8x")
    print("    4. Disaggregation (Dynamo: ~1.6-2.5x)             ~2.0x")
    print("  Product ~14x raw, closer to 7x after overhead and real-traffic alpha.")
    print("  Validate NVFP4 quality before migrating reasoning-heavy workloads.")


if __name__ == "__main__":
    main()