Gradient Clipping and Mixed Precision
The optimizer and schedule from the previous lesson assume gradients are sane. They usually are not. A single bad batch can spike the gradient norm by three orders of magnitude. Mixed-precision training amplifies this by introducing FP16 overflow on the loss side. This lesson builds the two safety belts that production training cannot ship without: gradient clipping to a configured global L2 norm, and a mixed-precision loop with autocast and GradScaler that detects NaN and Inf, skips the step cleanly, and logs the scaling factor for forensics.
Learning Objectives
- Compute the global L2 norm over all parameter gradients and clip in place when it exceeds a configured threshold.
- Wrap a training step in autocast plus a GradScaler so FP16 forward and backward passes survive overflow.
- Detect NaN and Inf in the loss or gradient, skip the optimizer step, and log the skip.
- Report the GradScaler's scaling factor every step so a long sequence of skips is visible immediately.
The Problem
A training run that ran clean yesterday produces a loss curve that goes vertical at step 8,217. The culprit is a single batch whose gradient norm is 4,200, twenty times the previous peak. Without clipping the optimizer applies a step that resets every learning the model had done in the previous hour. With a global L2 clip at norm 1.0, the same batch contributes a unit-norm update; the loss stays on its trend line; the run survives.
Mixed-precision training pushes throughput by 2-3x by computing the forward pass and most of the backward pass in FP16. The cost is that FP16 has a narrow exponent range. A typical gradient that overflows in FP16 evaluates to Inf, which propagates through subsequent layers as NaN, which sets every weight to NaN at the next optimizer step. PyTorch's GradScaler solves this by multiplying the loss by a large scaling factor before the backward pass and dividing the gradients by the same factor before the optimizer step. If any gradient is Inf or NaN at unscale time, the scaler skips the step and halves the scaling factor; if the previous N steps were clean, the scaler doubles the factor. Over the course of training the factor finds the highest value the FP16 range allows.
The build problem is wiring the two correctly. Clip before unscale and the threshold is on scaled gradients; clip after unscale and the order of operations on the GradScaler matters. The right order is: scaler.scale(loss).backward()then scaler.unscale_(optimizer)then clip_grad_norm_then scaler.step(optimizer)then scaler.update(). Any other order produces a silently broken loop.
The Concept
Global L2 norm
The global L2 norm is the Euclidean norm of the concatenated gradient vector, not the per-parameter norm. PyTorch implements this as torch.nn.utils.clip_grad_norm_(parameters, max_norm). The function returns the pre-clip norm so the lesson can log both the natural and the clipped value, which is necessary for the "we are clipping at every step" diagnosis.
autocast and GradScaler
torch.amp.autocast(device_type) is the context manager that selectively runs eligible operations (most matmul-class operations) in FP16. torch.amp.GradScaler(device_type) is the helper that scales the loss before backward and inverse-scales the gradients before the optimizer step. The two are designed together; using one without the other is a configuration error the test should catch.
The lesson uses CPU autocast because that is what runs in CI; the same pattern transfers verbatim to CUDA by changing device_type="cpu" to device_type="cuda". The GradScaler on CPU is a stub (CPU autocast already operates in BF16 by default and does not need loss scaling), but the lesson includes the call sites so the wiring is identical to the GPU loop.
NaN and Inf detection
The detection happens in two places. First, the loss itself is checked with torch.isfinite before backward; an Inf or NaN loss does not produce useful gradients and is skipped without entering the optimizer. Second, after scaler.unscale_(optimizer) the lesson scans the unscaled gradients with has_non_finite_grad(...) and treats any Inf or NaN as a skip. The two checks together cover both the forward-pass and the backward-pass failure modes.
Scaling factor diagnostics
The scaling factor is the GradScaler's internal state. Every step the lesson reads scaler.get_scale() and logs it next to the learning rate and gradient norm. A healthy run shows the scaling factor climbing in powers of two until it saturates near 2^17 or 2^18. A misbehaving run shows the factor oscillating between high and low values, which is the signal that the model's gradients are sometimes in range and sometimes not. The diagnostic is invisible without logging.
Build It
code/main.py implements:
clip_global_l2_norm- a wrapper aroundtorch.nn.utils.clip_grad_norm_that returns both the pre-clip and post-clip norm.has_non_finite_grad- a helper that scans gradients for NaN and Inf.AmpTrainState- wraps a model, anAdamWoptimizer, a GradScaler, and an autocast device. Exposes astep(inputs, targets)that runs the full clipping, scaling, and skip-on-NaN pipeline.StepLogandSkipLog- structured per-step records.- A demo that trains a small
nn.Linearmodel for 20 steps, injects an Inf into the gradient on step 5 to exercise the skip path, and prints the resulting log.
Run it:
python3 code/main.pyThe script exits zero and prints a per-step log with each row tagged STEP or SKIP; at least one row is a SKIP.
Production Patterns
Four patterns elevate the loop to a production training step.
Skip counter as an alert, not a log line. A handful of skipped steps per training run is healthy. Hundreds of skips per epoch are a hard alert: the model is in a regime FP16 cannot hold and the loop is silently failing. The lesson tracks a 1,000-step rolling skip rate and would, in production, page on a rate above 5 percent.
Clip threshold lives in the config. max_norm = 1.0 is the modern default for language-model training. Sweep it on a small model first; larger thresholds let the model recover from genuinely difficult batches; smaller thresholds bound the worst case at the cost of a noisier loss curve. The threshold belongs in the same YAML or JSON config as the schedule from lesson 44.
Norm log goes to a CSV with the schedule. The CSV columns are step, lr, grad_l2_pre_clip, grad_l2_post_clip, loss, skipped, skip_reason, scaler_scale. A reviewer who opens the file sees the schedule, the gradient story, the scaling factor, and the skip outcome (with its reason) in one row. Splitting the columns across files is a recipe for misaligned analyses.
scaler.update() runs every step, even on skip. On a clean step the scaler reads its no-inf counter, increments it, and possibly doubles the factor. On a skipped step the scaler halves the factor and resets the counter. Forgetting update() on the skip path is the bug that produces "the scaling factor never changed."
Use It
Production patterns:
- Autocast device matches optimizer device.
torch.amp.autocast(device_type="cuda")for GPU training;torch.amp.autocast(device_type="cpu")for CPU. Mixing devices produces a silent type error that surfaces as a loss curve that looks fine but a model that is not learning. - Loss check before backward.
torch.isfinite(loss).all()is one tensor reduction; the cost is negligible and the savings on a NaN loss are an entire training step. Always run it. set_to_none=Trueinzero_grad. Sets gradients toNoneinstead of zero, which lets the optimizer skip computation for unaffected parameter groups. The setting is a free throughput improvement and a slight bug-surface reduction.
Ship It
outputs/skill-clip-amp.md would, on a real project, describe which clip threshold and autocast device the training step uses, where the per-step CSV lives in version control, and what the production skip-rate alert threshold is. This lesson ships the engine.
Exercises
- Replace the synthetic Inf injection with a real loss spike (multiply one batch's target by 1e8) and verify the skip path triggers.
- Add a
--bf16mode that switches autocast to BF16 instead of FP16. BF16 has a wider exponent range than FP16 and rarely needs loss scaling; verify the skip rate drops to zero on the same demo. - Add a unit test that the gradient-clip wrapper returns the pre-clip and post-clip norm correctly when no clipping occurs.
- Add a rolling-window skip-rate computation and a CLI flag that fails the run if the rate exceeds a configured threshold for 100 consecutive steps.
- Wire the loop to write the canonical CSV (
step, lr, grad_l2_pre_clip, grad_l2_post_clip, loss, skipped, skip_reason, scaler_scale) and confirm the file survives a Ctrl-C by flushing after every row.
Key Terms
| Term | What people say | What it actually means |
|---|---|---|
| Global L2 norm | "Clip target" | Euclidean norm of the concatenated gradient vector across all trainable parameters |
| autocast | "Mixed precision" | Selective FP16 (or BF16) execution of eligible operations inside a with block |
| GradScaler | "Loss scaler" | Helper that multiplies the loss before backward and inverse-scales gradients before the optimizer step |
| Skip | "Bad step" | An optimizer step refused because the gradient or loss was non-finite; the scaler halves the factor |
| Scaling factor | "Scaler state" | The GradScaler's current multiplier; doubles after clean stretches and halves on every skip |
Further Reading
- Micikevicius et al., Mixed Precision Training (arXiv 1710.03740) - the original loss-scaling proposal
- Pascanu, Mikolov, Bengio, On the difficulty of training recurrent neural networks (arXiv 1211.5063) - the gradient-clipping reference paper
- PyTorch torch.amp.GradScaler - the scaler API this lesson wraps
- PyTorch torch.nn.utils.clip_grad_norm_ - the clipping primitive this lesson uses
- Phase 19 · 42 - the downloader whose corpus feeds the loop
- Phase 19 · 43 - the dataloader the loop consumes
- Phase 19 · 44 - the schedule this loop composes with
Check what you learned
6 questions. Answer each one to see why.
What problem does global L2 gradient clipping solve?
The full code
Every file for this lesson. Copy it, run it, then change it.
"""Gradient clipping and mixed-precision training step.
Implements:
- clip_global_l2_norm, a wrapper around torch.nn.utils.clip_grad_norm_ that
returns both the pre-clip norm and an explicit post-clip norm.
- has_non_finite_grad, a helper that scans gradients for NaN and Inf.
- AmpTrainState, a training-step orchestrator that wires an AdamW optimizer,
autocast, and GradScaler into one safe step.
- StepLog and SkipLog, structured per-step records.
The demo at the bottom trains a small torch.nn.Linear model for 20 steps and
injects a non-finite gradient on a specific step to exercise the skip path.
Run: python3 code/main.py
"""
from __future__ import annotations
import csv
import math
import sys
from dataclasses import asdict, dataclass, field
from pathlib import Path
from typing import Callable, Iterable
try:
import torch
from torch import nn
except ImportError as exc:
raise SystemExit(
"torch is required for this lesson. Install with: pip install torch"
) from exc
DEFAULT_MAX_NORM = 1.0
DEFAULT_DEVICE = "cpu"
NORM_TYPE = 2.0
@dataclass
class StepLog:
"""One row of the per-step training log."""
step: int
lr: float
grad_l2_pre_clip: float
grad_l2_post_clip: float
loss: float
skipped: bool
skip_reason: str
scaler_scale: float
def to_csv_row(self) -> list[str]:
return [
str(self.step),
f"{self.lr:.10f}",
f"{self.grad_l2_pre_clip:.10f}",
f"{self.grad_l2_post_clip:.10f}",
f"{self.loss:.10f}",
"1" if self.skipped else "0",
self.skip_reason,
f"{self.scaler_scale:.6f}",
]
@dataclass
class SkipLog:
"""Standalone record of a skipped step, for alerting and forensics."""
step: int
reason: str
pre_clip_norm: float
loss: float
scaler_scale: float
def has_non_finite_grad(parameters: Iterable[torch.nn.Parameter]) -> bool:
"""Return True if any gradient contains a NaN or Inf."""
for param in parameters:
if param.grad is None:
continue
grad = param.grad.detach()
if not torch.isfinite(grad).all().item():
return True
return False
def compute_global_l2_norm(parameters: Iterable[torch.nn.Parameter]) -> float:
"""Compute the Euclidean norm over all gradients without clipping."""
squared_sum = 0.0
for param in parameters:
if param.grad is None:
continue
grad = param.grad.detach()
squared_sum += float(grad.pow(2).sum().item())
return math.sqrt(squared_sum)
def clip_global_l2_norm(
parameters: list[torch.nn.Parameter],
max_norm: float,
) -> tuple[float, float]:
"""Clip gradients in place to max_norm and return (pre_clip, post_clip).
Returns (pre_clip, post_clip). When pre_clip <= max_norm the gradients are
untouched and post_clip == pre_clip. When pre_clip > max_norm the gradients
are scaled by max_norm / pre_clip and post_clip == max_norm.
"""
if max_norm <= 0:
raise ValueError("max_norm must be positive")
pre_clip = compute_global_l2_norm(parameters)
if not math.isfinite(pre_clip):
return pre_clip, pre_clip
if pre_clip <= max_norm:
return pre_clip, pre_clip
scale = max_norm / (pre_clip + 1e-12)
for param in parameters:
if param.grad is not None:
param.grad.detach().mul_(scale)
return pre_clip, max_norm
class AmpTrainState:
"""Training step with mixed precision and gradient clipping.
Wires together a model, an AdamW optimizer, a GradScaler, and an autocast
device. Exposes step(inputs, targets) which:
1. Forward pass under autocast.
2. Loss finiteness check; non-finite loss skips backward.
3. Backward through scaler.scale(loss).
4. scaler.unscale_(optimizer).
5. Gradient finiteness check; non-finite grad skips optimizer step.
6. Clip to max_norm.
7. scaler.step(optimizer); scaler.update().
"""
def __init__(
self,
model: nn.Module,
lr: float = 1e-2,
max_norm: float = DEFAULT_MAX_NORM,
device_type: str = DEFAULT_DEVICE,
weight_decay: float = 0.01,
amp_dtype: torch.dtype | None = None,
) -> None:
if max_norm <= 0:
raise ValueError("max_norm must be positive")
if device_type not in ("cpu", "cuda"):
raise ValueError(f"device_type must be 'cpu' or 'cuda', got {device_type}")
self.model = model
self.max_norm = max_norm
self.device_type = device_type
self.optimizer = torch.optim.AdamW(
model.parameters(),
lr=lr,
weight_decay=weight_decay,
)
scaler_enabled = device_type == "cuda"
self.scaler = torch.amp.GradScaler(device_type, enabled=scaler_enabled)
if amp_dtype is None:
amp_dtype = torch.bfloat16 if device_type == "cpu" else torch.float16
self.amp_dtype = amp_dtype
self.global_step = 0
self._log: list[StepLog] = []
self._skip_log: list[SkipLog] = []
self._loss_fn: Callable[[torch.Tensor, torch.Tensor], torch.Tensor] = nn.functional.mse_loss
@property
def log(self) -> list[StepLog]:
return list(self._log)
@property
def skip_log(self) -> list[SkipLog]:
return list(self._skip_log)
@property
def skip_count(self) -> int:
return len(self._skip_log)
def set_loss_fn(self, fn: Callable[[torch.Tensor, torch.Tensor], torch.Tensor]) -> None:
self._loss_fn = fn
def set_lr(self, lr: float) -> None:
for group in self.optimizer.param_groups:
group["lr"] = lr
def _current_lr(self) -> float:
return float(self.optimizer.param_groups[0]["lr"])
def step(
self,
inputs: torch.Tensor,
targets: torch.Tensor,
gradient_corruptor: Callable[[nn.Module], None] | None = None,
) -> StepLog:
"""Run one training step with optional gradient corruption for testing.
`gradient_corruptor` lets the demo inject a non-finite gradient after
backward and before the unscale step. Production callers leave it as
None; tests pass a closure that writes Inf into one parameter's grad.
"""
self.model.train()
self.optimizer.zero_grad(set_to_none=True)
with torch.amp.autocast(device_type=self.device_type, dtype=self.amp_dtype):
predictions = self.model(inputs)
loss = self._loss_fn(predictions, targets)
if not torch.isfinite(loss).all().item():
# Skip without touching scaler.update(): we never called
# scaler.scale(loss).backward() for this step, so calling update()
# here would violate GradScaler's required call ordering.
return self._record_skip(
loss_value=float(loss.detach().cpu().item()),
reason="non_finite_loss",
pre_clip=0.0,
update_scaler=False,
)
self.scaler.scale(loss).backward()
if gradient_corruptor is not None:
gradient_corruptor(self.model)
self.scaler.unscale_(self.optimizer)
if has_non_finite_grad(self.model.parameters()):
scale_before = float(self.scaler.get_scale())
self.scaler.update()
record = StepLog(
step=self.global_step,
lr=self._current_lr(),
grad_l2_pre_clip=float("inf"),
grad_l2_post_clip=float("inf"),
loss=float(loss.detach().item()),
skipped=True,
skip_reason="non_finite_grad",
scaler_scale=scale_before,
)
self._log.append(record)
self._skip_log.append(
SkipLog(
step=self.global_step,
reason="non_finite_grad",
pre_clip_norm=float("inf"),
loss=float(loss.detach().item()),
scaler_scale=scale_before,
)
)
self.global_step += 1
return record
pre_clip, post_clip = clip_global_l2_norm(list(self.model.parameters()), self.max_norm)
self.scaler.step(self.optimizer)
self.scaler.update()
record = StepLog(
step=self.global_step,
lr=self._current_lr(),
grad_l2_pre_clip=pre_clip,
grad_l2_post_clip=post_clip,
loss=float(loss.detach().item()),
skipped=False,
skip_reason="",
scaler_scale=float(self.scaler.get_scale()),
)
self._log.append(record)
self.global_step += 1
return record
def _record_skip(
self,
loss_value: float,
reason: str,
pre_clip: float,
update_scaler: bool = True,
) -> StepLog:
record = StepLog(
step=self.global_step,
lr=self._current_lr(),
grad_l2_pre_clip=pre_clip,
grad_l2_post_clip=pre_clip,
loss=loss_value,
skipped=True,
skip_reason=reason,
scaler_scale=float(self.scaler.get_scale()),
)
self._log.append(record)
self._skip_log.append(
SkipLog(
step=self.global_step,
reason=reason,
pre_clip_norm=pre_clip,
loss=loss_value,
scaler_scale=float(self.scaler.get_scale()),
)
)
self.global_step += 1
if update_scaler:
self.scaler.update()
return record
def rolling_skip_rate(log: Iterable[StepLog], window: int = 1000) -> list[float]:
"""Return the rolling skip rate over the last `window` steps for each step."""
if window <= 0:
raise ValueError("window must be positive")
rows = list(log)
rates: list[float] = []
skipped: list[int] = []
for row in rows:
skipped.append(1 if row.skipped else 0)
if len(skipped) > window:
skipped = skipped[-window:]
rates.append(sum(skipped) / len(skipped))
return rates
def write_step_log_csv(log: Iterable[StepLog], path: Path) -> None:
"""Write the canonical training-step CSV.
Columns: step, lr, grad_l2_pre_clip, grad_l2_post_clip, loss, skipped,
skip_reason, scaler_scale.
"""
path = Path(path)
path.parent.mkdir(parents=True, exist_ok=True)
with path.open("w", encoding="utf-8", newline="") as fh:
writer = csv.writer(fh)
writer.writerow(
[
"step",
"lr",
"grad_l2_pre_clip",
"grad_l2_post_clip",
"loss",
"skipped",
"skip_reason",
"scaler_scale",
]
)
for row in log:
writer.writerow(row.to_csv_row())
def build_toy_model(
in_dim: int = 16,
out_dim: int = 4,
seed: int = 7,
) -> tuple[nn.Module, torch.Tensor, torch.Tensor]:
torch.manual_seed(seed)
model = nn.Sequential(nn.Linear(in_dim, 32), nn.GELU(), nn.Linear(32, out_dim))
inputs = torch.randn(8, in_dim)
targets = torch.randn(8, out_dim)
return model, inputs, targets
def inject_inf_into_first_grad(model: nn.Module) -> None:
"""Test-only: write +Inf into the first parameter's gradient."""
for param in model.parameters():
if param.grad is not None:
param.grad.data[...] = float("inf")
return
def run_demo() -> int:
"""Train for 20 steps and inject a non-finite gradient on a known step."""
model, inputs, targets = build_toy_model()
state = AmpTrainState(model=model, lr=1e-2, max_norm=1.0, device_type="cpu")
for index in range(20):
corruptor: Callable[[nn.Module], None] | None = None
if index == 5:
corruptor = inject_inf_into_first_grad
record = state.step(inputs, targets, gradient_corruptor=corruptor)
marker = "SKIP" if record.skipped else "STEP"
print(
f"{marker} step={record.step:>3} lr={record.lr:.6f} "
f"pre_clip={record.grad_l2_pre_clip:>10.6f} "
f"post_clip={record.grad_l2_post_clip:>10.6f} "
f"loss={record.loss:.6f} scale={record.scaler_scale:.1f} "
f"reason={record.skip_reason or '-'}"
)
print()
print(
f"skip_count={state.skip_count} "
f"final_skip_rate={rolling_skip_rate(state.log, window=10)[-1]:.4f}"
)
return 0
if __name__ == "__main__":
sys.exit(run_demo())
Most people stop here. Don't be most people.
The next 42 lessons are where Phase 19 turns from understanding into ability: the harder builds, the production patterns and the quizzes that prove you've actually got it. Unlock the full AI Expert Masterclass: all 542 expert lessons, every line of code, every phase.
- 44Gradient Accumulation
- 45Checkpoint Save and Resume
- 46Distributed Data Parallel and FSDP from Scratch
- 47Language Model Evaluation Harness
- + 38 more in this phase, and 542 across the whole masterclass