Get Started

Capstone 07, End-to-End Fine-Tuning Pipeline (Data to SFT to DPO to Serve)

An 8B model trained on your own data, DPO-aligned on your own preferences, quantized, speculative-decoded, and served at measurable $/1M tokens. The 2026 open stack is Axolotl v0.8, TRL 0.15, Unsloth for iteration, GPTQ/AWQ/GGUF for quantization, vLLM 0.7 with EAGLE-3 for serving. The capstone is to run the whole pipeline reproducibly, YAML in, served endpoint out, and publish a model card under the 2026 Model Openness Framework.

35 hoursCapstonePython (pipeline), YAML (configs), Bash (scripts)Free lesson

Problem

Every serious AI team in 2026 keeps a fine-tuning pipeline on tap. Not because they ship a frontier base model, but because downstream adaptation, domain SFT, DPO against labeled preferences, distilled drafts for speculative decoding, serving with EAGLE-3, is where the measurable wins live. Axolotl v0.8 handles multi-GPU SFT configs. TRL 0.15 handles DPO and GRPO. Unsloth gets you fast single-GPU iteration. vLLM 0.7 with EAGLE-3 pushes decode throughput 2-3x without quality loss. The tooling works; the craft is in the YAMLs, the data hygiene, and the eval discipline.

You will run an 8B base (Llama 3.3, Qwen3, or Gemma 3) through SFT then DPO on task-specific data, quantize for serving, and measure gains against lm-evaluation-harness, RewardBench-2, MT-Bench-v2, and MMLU-Pro. You will produce a model card under the 2026 Model Openness Framework. The point is reproducibility, one command reruns the whole pipeline end to end.

Concept

The pipeline has five stages. Data: dedup (MinHash / Datatrove), quality filter (Nemotron-CC style classifier), PII scrub, split-hygiene check against public benchmark contamination. SFT: Axolotl YAML, ZeRO-3 on 8xH100, cosine schedule, packed sequences, 2-3 epochs. DPO or GRPO: TRL config, 1 epoch, preference pairs either human-labeled or model-judged, beta tuning. Quantize: GPTQ + AWQ + GGUF for deployment flexibility. Serve: vLLM 0.7 with EAGLE-3 speculative heads (or SGLang with SpecForge), K8s deployment, HPA on queue-wait.

Ablations are the deliverable: SFT-only vs SFT+DPO vs SFT+GRPO on three task-specific benchmarks. Serving metrics: tokens/s at batch 1 / 8 / 32, EAGLE-3 acceptance rate, $/1M tokens. Safety eval: Llama Guard 4 pass rate. Model card: bias evaluations, reproducibility seeds, data licensing.

Architecture

Code
raw data (HF datasets + internal)
    |
    v
Datatrove dedup + Nemotron-CC quality filter + PII scrub
    |
    v
split hygiene (MMLU-Pro contamination check)
    |
    v
Axolotl SFT config (YAML)  ---> 8xH100, ZeRO-3
    |
    v
TRL DPO / GRPO config       ---> 4xH100, 1 epoch
    |
    v
GPTQ + AWQ + GGUF quantize
    |
    v
vLLM 0.7 + EAGLE-3 speculative decoding
    |
    v
K8s deployment, HPA on queue-wait
    |
    v
lm-eval-harness + RewardBench-2 + MT-Bench-v2 + MMLU-Pro
    |
    v
model card (2026 MOF) + safety eval (Llama Guard 4)

Stack

  • Data: Datatrove for dedup, Nemotron-CC classifier for quality, Presidio for PII
  • Base: Llama 3.3 8B, Qwen3 14B, or Gemma 3 12B
  • SFT: Axolotl v0.8 with ZeRO-3, Flash Attention 3, packed sequences
  • Preference tuning: TRL 0.15 for DPO or GRPO; Unsloth for single-GPU iteration
  • Quantization: GPTQ (Marlin), AWQ, GGUF via llama.cpp
  • Serving: vLLM 0.7 with EAGLE-3 speculative decoding (or SGLang 0.4 + SpecForge)
  • Eval: lm-evaluation-harness, RewardBench-2, MT-Bench-v2, MMLU-Pro
  • Safety eval: Llama Guard 4, ShieldGemma-2
  • Infrastructure: Kubernetes + NVIDIA device plugin, HPA on queue-wait metric
  • Observability: W&B for training, Langfuse for inference

Build It

  1. Data pipeline. Run Datatrove dedup on raw corpus. Apply Nemotron-CC-style quality classifier. Presidio scrubs PII. Write train/val splits with explicit seed.
  2. Contamination check. For every validation split, compute MinHash against MMLU-Pro, MT-Bench-v2, RewardBench-2 test sets. Reject any overlap.
  3. Axolotl SFT. YAML with ZeRO-3, FA3, sequence packing. 2-3 epochs on 8xH100. Log to W&B.
  4. TRL DPO / GRPO. Take the SFT checkpoint, run one epoch of DPO on preference pairs (or GRPO with a verifiable reward on math/code). Sweep beta.
  5. Quantize. Produce three quants: GPTQ-INT4-Marlin, AWQ-INT4, GGUF-Q4_K_M for llama.cpp. Record size and nominal throughput.
  6. Serve with speculative decoding. vLLM 0.7 config with EAGLE-3 draft heads trained via Red Hat Speculators. Measure acceptance rate and tail latency at batch 1 / 8 / 32. Report $/1M tokens vs Anthropic / OpenAI on the same eval.
  7. Eval matrix. Run lm-eval-harness, RewardBench-2, MT-Bench-v2, MMLU-Pro on base, SFT-only, SFT+DPO, SFT+GRPO. Produce a table.
  8. Safety eval. Llama Guard 4 pass rate on the dev set. ShieldGemma-2 output filter.
  9. Model card. MOF 2026 template: data, training, eval, safety, license, reproducibility section with YAMLs and commit SHAs.

Use It

Code
$ ./pipeline.sh config/llama3.3-8b-domainX.yaml
[data]    300k deduped, 12k filtered, 280k accepted (seed=7)
[SFT]     3 epochs, 8xH100, 6h12m, val loss 1.42 -> 1.03
[DPO]     1 epoch, beta=0.08, 4xH100, 1h40m
[quant]   GPTQ-INT4 4.6 GB, AWQ-INT4 4.8 GB, GGUF-Q4_K_M 5.1 GB
[serve]   vLLM 0.7, EAGLE-3 acceptance 0.74, p99 126ms @ bs=8
[eval]    MMLU-Pro +3.2, MT-Bench-v2 +0.41, RewardBench-2 +0.08
[card]    model-card.md generated under 2026 MOF

Ship It

outputs/skill-finetuning-pipeline.md describes the deliverable. A single command runs data through SFT through DPO through quant through serve through eval, and emits a model card + the served endpoint.

WeightCriterionHow it is measured
25Eval delta vs baseMeasured gain on target tasks (MMLU-Pro, MT-Bench-v2, task-specific)
20Pipeline reproducibilityOne command reruns end to end with identical seeds
20Data hygieneDedup rate, PII scrub coverage, contamination check green
20Serving efficiencytokens/s at bs=1/8/32, EAGLE-3 acceptance rate, $/1M tokens
15Model card + safety eval2026 MOF completeness + Llama Guard 4 pass rate
100

Exercises

  1. Run SFT-only vs SFT+DPO vs SFT+GRPO on the same task-specific benchmark. Report which preference method wins and by how much.
  2. Swap Llama 3.3 8B for Qwen3 14B. Measure the $/1M tokens at matched quality.
  3. Measure EAGLE-3 acceptance rate on domain data vs generic ShareGPT. Report the delta and what it means for latency budgets.
  4. Inject 1% of contamination (leak MMLU-Pro answers into training data) and rerun eval. Watch MMLU-Pro accuracy jump unrealistically. Build a contamination-check CI gate that catches this.
  5. Add LoRA SFT as an alternative to full fine-tune. Measure the quality gap at 10x lower memory.

Key Terms

TermWhat people sayWhat it actually means
Axolotl"SFT trainer"Unified YAML-driven trainer for SFT, DPO, and distillation
TRL"Preference tuner"Hugging Face library for DPO, GRPO, PPO on LLMs
GRPO"Group-relative policy optimization"DeepSeek R1's RL recipe with verifiable rewards
EAGLE-3"Speculative decoding draft"Draft heads that predict N tokens ahead; vLLM verifies with target model
MOF"Model Openness Framework"2026 standard for grading model releases on data, code, license
Contamination check"Split hygiene"MinHash-based detection of test-set leakage into training
Acceptance rate"EAGLE / MTP metric"Fraction of drafted tokens the target model accepts

Further Reading

Check what you learned

7 questions. Answer each one to see why.

Question 1 of 70 correct

What does the contamination check guard against during data preparation?

The full code

Every file for this lesson. Copy it, run it, then change it.

main.py
"""End-to-end fine-tuning pipeline orchestrator scaffold.

The hard architectural primitive is a reproducible pipeline DAG: data hygiene
-> SFT -> preference tuning -> quantization -> serving -> eval -> model card,
where each stage is declaratively configured (YAML-ish dict here) and each
stage consumes the previous stage's artifact by content hash. This scaffold
models the DAG, the artifact manifest, and the contamination check.

Run:  python main.py
"""

from __future__ import annotations

import hashlib
import json
import time
from dataclasses import dataclass, field
from typing import Callable


# ---------------------------------------------------------------------------
# artifact + manifest  --  content-hashed bookkeeping
# ---------------------------------------------------------------------------

@dataclass
class Artifact:
    name: str
    kind: str         # "dataset" | "checkpoint" | "quant" | "endpoint" | "report"
    payload: dict
    produced_by: str
    produced_at: float = field(default_factory=time.time)

    def content_hash(self) -> str:
        blob = json.dumps(self.payload, sort_keys=True, default=str).encode()
        return hashlib.sha256(blob).hexdigest()[:12]


@dataclass
class Manifest:
    artifacts: dict[str, Artifact] = field(default_factory=dict)

    def add(self, a: Artifact) -> None:
        self.artifacts[a.name] = a

    def get(self, name: str) -> Artifact:
        return self.artifacts[name]

    def summary(self) -> list[tuple[str, str, str, str]]:
        return [(a.name, a.kind, a.content_hash(), a.produced_by)
                for a in self.artifacts.values()]


# ---------------------------------------------------------------------------
# stages  --  each returns a new Artifact given prior manifest and config
# ---------------------------------------------------------------------------

Stage = Callable[[Manifest, dict], Artifact]


def stage_data(m: Manifest, cfg: dict) -> Artifact:
    raw_n = cfg.get("raw_examples", 300_000)
    dedup_ratio = 0.94
    qual_ratio = 0.91
    pii_ratio = 0.995
    kept = int(raw_n * dedup_ratio * qual_ratio * pii_ratio)
    return Artifact("dataset", "dataset", {
        "raw_examples": raw_n,
        "after_dedup": int(raw_n * dedup_ratio),
        "after_quality": int(raw_n * dedup_ratio * qual_ratio),
        "after_pii_scrub": kept,
        "seed": cfg.get("seed", 7),
    }, produced_by="Datatrove+Nemotron-CC+Presidio")


def stage_contamination(m: Manifest, cfg: dict) -> Artifact:
    ds = m.get("dataset")
    overlap = []
    for bench in ("MMLU-Pro", "MT-Bench-v2", "RewardBench-2"):
        # simulated MinHash check; real pipeline uses Datatrove MinHashLSH
        overlap.append({"bench": bench, "overlap_examples": 0})
    return Artifact("contamination_check", "report", {
        "dataset_hash": ds.content_hash(),
        "overlaps": overlap,
        "status": "clean" if all(o["overlap_examples"] == 0 for o in overlap) else "dirty",
    }, produced_by="minhash-lsh")


def stage_sft(m: Manifest, cfg: dict) -> Artifact:
    ds = m.get("dataset")
    return Artifact("sft_checkpoint", "checkpoint", {
        "base": cfg["base_model"],
        "dataset_hash": ds.content_hash(),
        "epochs": 3,
        "val_loss": 1.03,
        "hours": 6.2,
        "gpus": 8,
    }, produced_by="axolotl v0.8 + ZeRO-3")


def stage_dpo(m: Manifest, cfg: dict) -> Artifact:
    sft = m.get("sft_checkpoint")
    return Artifact("dpo_checkpoint", "checkpoint", {
        "from": sft.content_hash(),
        "epochs": 1,
        "beta": 0.08,
        "hours": 1.7,
    }, produced_by="trl 0.15 DPO")


def stage_quantize(m: Manifest, cfg: dict) -> Artifact:
    ckpt = m.get("dpo_checkpoint")
    return Artifact("quants", "quant", {
        "from": ckpt.content_hash(),
        "gptq_int4_gb": 4.6,
        "awq_int4_gb": 4.8,
        "gguf_q4_km_gb": 5.1,
    }, produced_by="gptq+awq+llama.cpp")


def stage_serve(m: Manifest, cfg: dict) -> Artifact:
    quants = m.get("quants")
    return Artifact("endpoint", "endpoint", {
        "backend": "vLLM 0.7 + EAGLE-3",
        "quant": "GPTQ-INT4-Marlin",
        "eagle_acceptance": 0.74,
        "p99_bs8_ms": 126,
        "tokens_per_sec_bs32": 6400,
        "dollars_per_mtokens": 0.28,
    }, produced_by="vllm+speculators")


def stage_eval(m: Manifest, cfg: dict) -> Artifact:
    ckpt = m.get("dpo_checkpoint")
    return Artifact("eval_report", "report", {
        "from": ckpt.content_hash(),
        "mmlu_pro_delta": 3.2,
        "mt_bench_v2_delta": 0.41,
        "rewardbench2_delta": 0.08,
        "llama_guard_4_pass": 0.987,
    }, produced_by="lm-eval-harness")


def stage_model_card(m: Manifest, cfg: dict) -> Artifact:
    return Artifact("model_card", "report", {
        "standard": "MOF 2026",
        "data_license_declared": True,
        "training_config_hash": m.get("sft_checkpoint").content_hash(),
        "eval_attached": True,
        "safety_attached": True,
        "reproducibility_command": "./pipeline.sh config/llama3.3-8b-domainX.yaml",
    }, produced_by="mof-template")


# ---------------------------------------------------------------------------
# DAG orchestrator  --  runs stages in order, snapshots manifest each step
# ---------------------------------------------------------------------------

PIPELINE: list[tuple[str, Stage]] = [
    ("data", stage_data),
    ("contamination", stage_contamination),
    ("sft", stage_sft),
    ("dpo", stage_dpo),
    ("quantize", stage_quantize),
    ("serve", stage_serve),
    ("eval", stage_eval),
    ("model_card", stage_model_card),
]


def run_pipeline(cfg: dict) -> Manifest:
    m = Manifest()
    for name, stage_fn in PIPELINE:
        print(f"[{name:14s}] running...")
        art = stage_fn(m, cfg)
        m.add(art)
        print(f"[{name:14s}] -> artifact '{art.name}' hash={art.content_hash()}")
    return m


def main() -> None:
    cfg = {
        "base_model": "llama-3.3-8b",
        "raw_examples": 300_000,
        "seed": 7,
        "dpo_beta": 0.08,
    }
    print("=== fine-tuning pipeline run ===")
    m = run_pipeline(cfg)
    print()
    print("=== manifest ===")
    for name, kind, h, by in m.summary():
        print(f"  {name:18s} {kind:10s} {h} by {by}")
    print()
    print("=== eval report ===")
    print(json.dumps(m.get("eval_report").payload, indent=2))
    print()
    print("=== served endpoint ===")
    print(json.dumps(m.get("endpoint").payload, indent=2))


if __name__ == "__main__":
    main()