Get Started

Production Quantization, AWQ, GPTQ, GGUF K-quants, FP8, MXFP4/NVFP4

Quantization format is not a universal choice, it is a function of hardware, serving engine, and workload. GGUF Q4_K_M or Q5_K_M owns CPU and edge, delivered through llama.cpp and Ollama. GPTQ wins inside vLLM when you need multi-LoRA on the same base. AWQ with Marlin-AWQ kernels delivers ~741 tok/s on a 7B class model with the best Pass@1 at INT4, the 2026 default for datacenter production. FP8 stays the middle ground on Hopper, Ada, and Blackwell, near-lossless and widely supported. NVFP4 and MXFP4 (Blackwell microscaling) are aggressive and require per-block validation. Two traps bite teams: calibration dataset must match deployment domain, and KV cache is separate from weight quantization, the AWQ lesson "my model is 4 GB now" forgets the 10-30 GB KV cache at production batch sizes.

75 minutesLearnPython (stdlib, toy memory and throughput comparison across formats)Free lesson

Learning Objectives

  • Name the six production quantization formats and their sweet spots in 2026.
  • Pick a format given hardware (CPU vs GPU, Hopper vs Blackwell), engine (vLLM, TRT-LLM, llama.cpp), and workload (routine chat, reasoning, multi-LoRA).
  • Compute the weight memory saved and the KV cache left untouched for a chosen format.
  • Name the calibration-dataset pitfall that degrades quantized models on domain traffic.

The Problem

Quantization reduces memory and HBM bandwidth, which is exactly what decode needs. An FP16 70B model is 140 GB of weights. Quantize weights to INT4 (AWQ or GPTQ) and the model is 35 GB, fits in one H100 with room for KV cache, which matters because at 128 concurrent sequences with 2k context, KV cache alone is 20-30 GB.

But quantization is not free. Aggressive quantization degrades quality, especially on reasoning-heavy tasks. Different formats work with different engines. Different hardware supports different precisions natively. The 2026 format zoo is real and you cannot copy someone else's choice, you have to pick based on your stack.

The Concept

The six formats

FormatBitsSweet spotEngines
GGUF Q4_K_M / Q5_K_M4-5CPU, edge, laptopsllama.cpp, Ollama
GPTQ4-8Multi-LoRA on vLLMvLLM, TGI
AWQ4Datacenter GPU productionvLLM (Marlin-AWQ), TGI
FP88Hopper/Ada/Blackwell datacentervLLM, TRT-LLM, SGLang
MXFP44Blackwell multi-userTRT-LLM
NVFP44Blackwell multi-userTRT-LLM

GGUF, the CPU/edge default

GGUF is a file format, not a quantization scheme per se, it bundles K-quant variants (Q2_K, Q3_K_M, Q4_K_M, Q5_K_M, Q6_K, Q8_0) in one container. Q4_K_M and Q5_K_M are the production defaults, near-BF16 quality at 4-5 bits. Best choice for CPU or edge serving because llama.cpp is by far the fastest CPU inference engine.

Throughput penalty in vLLM: ~93 tok/s on 7B, the format is not optimized for GPU kernels. Use GGUF when the deployment target is CPU/edge. Not otherwise.

GPTQ, multi-LoRA in vLLM

GPTQ is a post-training quantization algorithm with a calibration pass. Marlin kernels make it fast on GPU (2.6x speedup vs non-Marlin GPTQ). ~712 tok/s on 7B.

The unique win: GPTQ-Int4 supports LoRA adapters in vLLM. If you are serving a base model plus 10-50 fine-tuned variants (each as a LoRA), GPTQ is your path. NVFP4 does not support LoRA yet as of early 2026.

AWQ, the datacenter GPU default

Activation-aware Weight Quantization. Protects the ~1% most-salient weights during quantization. Marlin-AWQ kernels: 10.9x speedup vs naive. ~741 tok/s on 7B, best Pass@1 among INT4 formats.

Pick AWQ for new GPU serving unless you need multi-LoRA (GPTQ) or aggressive Blackwell FP4 (NVFP4).

FP8, the reliable middle

8-bit floating point. Near-lossless. Widely supported. Hopper Tensor Cores accelerate FP8 natively. Blackwell inherits. FP8 is the safe 2026 default when quality is non-negotiable (reasoning, medical, code-gen). Memory savings are half of INT4 but quality risk is far lower.

MXFP4 / NVFP4, Blackwell aggressive

Microscaling FP4. Each block of weights has its own scale factor. Aggressive but hardware-accelerated on Blackwell Tensor Cores. Halve the bytes per token versus FP8, the economic win in Phase 17 · 07.

Caveats:

  • No LoRA support yet (early 2026).
  • Quality drop visible on reasoning-heavy workloads.
  • Validate on your eval set per model.

The calibration trap

AWQ and GPTQ require a calibration dataset, typically C4 or WikiText. For domain models (code, medical, legal), calibrating on generic web text lets the algorithm make wrong decisions about which weights to protect. Pass@1 on HumanEval can drop several points.

The fix: calibrate on in-domain data. Hundreds of domain samples is usually enough. Test on the eval set before shipping.

The KV cache trap

AWQ shrinks weights to 4 bits. KV cache is separate and stays at FP16/FP8. For a 70B model with AWQ:

  • Weights: ~35 GB (INT4 from 140 GB).
  • KV cache at 128 concurrent × 2k context: ~20 GB.
  • Activations: ~5 GB.
  • Total: ~60 GB, fits on H100 80GB.

Naively "I quantized my model to 4 GB" forgets the other 30-50 GB. Budget HBM holistically.

Separately, KV cache quantization (FP8 KV or INT8 KV) is a different choice with its own tradeoffs, it affects attention accuracy directly and is not a free win.

AWQ INT4 is hazardous for reasoning

Chain-of-thought, math, code-gen with long context, these suffer visibly from aggressive quantization. AWQ INT4 loses ~3-5 points on MATH. For reasoning-heavy workloads, ship FP8 or BF16; accept the memory cost.

2026 picking guide

  • CPU/edge serve: GGUF Q4_K_M. Done.
  • GPU serve, routine chat, no LoRA: AWQ.
  • GPU serve, multi-LoRA: GPTQ with Marlin.
  • Reasoning workload: FP8.
  • Blackwell datacenter, validated quality: NVFP4 + FP8 KV.
  • Ambiguous: run a 1,000-sample eval on each candidate format.

Use It

code/main.py computes memory footprint (weights + KV + activations) and relative throughput across the six formats for a range of model sizes. Shows where KV cache dominates, where weight compression pays, and where FP8 is the safe pick.

The Inside an LLM Serving GPU machine shows FP8 on an NVIDIA L4. The weights fall from 15.28 GB to 8.81 GB, the KV pool grows from 24,094 to 142,369 tokens, and the measured output at 64 users goes from 250.85 to 735.91 tokens per second.

Ship It

This lesson produces outputs/skill-quantization-picker.md. Given hardware, model size, workload type, and quality tolerance, picks a format and produces a calibration/validation plan.

Exercises

  1. Run code/main.py. For a 70B model at 128 concurrent with 2k context, compute the total HBM for each format. Which format lets you fit on one H100 80GB?
  2. You have a 7B coding model. Pick a format and justify. If you were wrong about quality tolerance, what is the recovery path?
  3. Compute the calibration-dataset size needed to calibrate AWQ for a medical domain model. Why is more data not always better?
  4. Read the Marlin-AWQ kernel paper or release notes. Explain in three sentences why AWQ hits 741 tok/s on 7B while raw GPTQ hits ~712.
  5. When does it make sense to combine AWQ weights with FP8 KV cache vs keeping KV at BF16?

Key Terms

TermWhat people sayWhat it actually means
GGUF"llama.cpp format"File format bundling K-quant variants; CPU/edge default
Q4_K_M"Q4 K M"4-bit K-quant medium; the production GGUF default
GPTQ"gee pee tee q"Post-train INT4 with calibration; supports LoRA in vLLM
AWQ"a w q"Activation-aware INT4; Marlin kernels; best Pass@1 at INT4
Marlin kernels"fast INT4 kernels"Custom CUDA kernels for INT4 on Hopper; 10x speedup
FP8"eight-bit float"Safe precision default on Hopper/Ada/Blackwell
MXFP4 / NVFP4"microscaling four"Blackwell 4-bit FP with per-block scale factors
Calibration dataset"cal data"Input text used to pick quantization parameters; must match domain
KV cache quantization"KV INT8"Separate choice from weights; affects attention accuracy

Further Reading

Check what you learned

6 questions. Answer each one to see why.

Question 1 of 60 correct

Which quantization format does the lesson call the production default for CPU and edge serving?

The full code

Every file for this lesson. Copy it, run it, then change it.

main.py
"""Toy quantization memory and throughput calculator — stdlib Python.

For a set of quantization formats and model sizes, compute:
  - weight memory
  - KV cache memory (separate, scales with concurrency and context)
  - activations memory (approximate)
  - relative decode throughput (memory-bandwidth-limited shape)

Formats are represented by effective weight bits and KV bits. Pedagogical.
"""

from __future__ import annotations

from dataclasses import dataclass


@dataclass
class Format:
    name: str
    weight_bits: float
    kv_bits: float
    engine: str
    notes: str


FORMATS = [
    Format("BF16 baseline (vLLM)",       16, 16, "vLLM",     "reference"),
    Format("GGUF Q5_K_M (llama.cpp)",     5, 16, "llama.cpp", "CPU/edge"),
    Format("GGUF Q4_K_M (llama.cpp)",     4, 16, "llama.cpp", "CPU/edge, default"),
    Format("GPTQ-Int4 + Marlin (vLLM)",   4, 16, "vLLM",     "multi-LoRA support"),
    Format("AWQ-Int4 + Marlin (vLLM)",    4, 16, "vLLM",     "best Pass@1 at INT4"),
    Format("FP8 (vLLM / TRT-LLM)",        8,  8, "multi",    "safe default reasoning"),
    Format("NVFP4 + FP8 KV (TRT-LLM)",    4,  8, "TRT-LLM",  "Blackwell aggressive"),
]


def memory_breakdown(params_b: float, fmt: Format,
                     concurrency: int = 128, ctx: int = 2048) -> dict:
    weight_gb = params_b * fmt.weight_bits / 8
    # KV cache approximation: num_layers * 2 * kv_heads * head_dim * ctx * bytes/element
    layers = 64 * (params_b / 70.0)**0.5
    kv_heads = 8
    head_dim = 128
    per_seq_kv_gb = layers * 2 * kv_heads * head_dim * ctx * (fmt.kv_bits / 8) / 1e9
    kv_total = per_seq_kv_gb * concurrency
    activations_gb = 0.05 * params_b       # rough constant
    return {
        "weight": weight_gb,
        "kv": kv_total,
        "act": activations_gb,
        "total": weight_gb + kv_total + activations_gb,
    }


def relative_throughput(fmt: Format) -> float:
    """Decode is memory-bandwidth-limited. Fewer weight bytes per token = higher throughput.
    Normalize to BF16 = 1.0."""
    return 16 / fmt.weight_bits


def gpu_check(total_gb: float) -> str:
    if total_gb <= 80:
        return "H100 80GB"
    if total_gb <= 141:
        return "H200 141GB"
    if total_gb <= 192:
        return "B200 192GB"
    return "MULTI-GPU"


def print_scenario(params_b: float, concurrency: int, ctx: int) -> None:
    print(f"Model: {params_b}B params  |  concurrency {concurrency}  |  ctx {ctx}")
    print("-" * 98)
    print(f"{'format':36} {'W GB':>7} {'KV GB':>7} {'Act GB':>7} "
          f"{'Total':>7} {'fits on':>14} {'rel tput':>10}")
    for f in FORMATS:
        m = memory_breakdown(params_b, f, concurrency, ctx)
        tput = relative_throughput(f)
        print(f"{f.name:36} {m['weight']:7.1f} {m['kv']:7.1f} {m['act']:7.1f} "
              f"{m['total']:7.1f} {gpu_check(m['total']):>14} {tput:10.2f}x")
    print()


def main() -> None:
    print("=" * 98)
    print("TOY QUANTIZATION CALCULATOR — memory and relative throughput by format")
    print("=" * 98)
    print()

    print_scenario(params_b=7, concurrency=128, ctx=2048)
    print_scenario(params_b=70, concurrency=128, ctx=2048)
    print_scenario(params_b=70, concurrency=256, ctx=8192)
    print_scenario(params_b=405, concurrency=128, ctx=2048)

    print("=" * 98)
    print("KEY FINDINGS")
    print("-" * 98)
    print("  1. KV cache grows linearly with concurrency x context.")
    print("     At 256 conc / 8k ctx on 70B, KV alone dwarfs weight savings.")
    print("  2. AWQ vs GPTQ = same 4-bit footprint; choice is about LoRA support and kernels.")
    print("  3. NVFP4 + FP8 KV stacks: shrink weights AND KV ; Blackwell-only.")
    print("  4. For reasoning workloads, FP8 is the safe default despite higher memory.")
    print("  5. GGUF wins on CPU ; ~93 tok/s in vLLM is not a bug, it is the wrong engine.")


if __name__ == "__main__":
    main()