Cosine LR with Linear Warmup
The learning-rate schedule is the second most important decision after the loss function. AdamW with a cosine decay and a linear warmup is the modern default for language-model training because it lets the model see a small effective step size during the brittle first thousand updates, ramps up to a configured peak, and decays smoothly back toward zero. This lesson builds that schedule, plots the curve over training steps, logs gradient norms next to the schedule, and proves the schedule honors warmup, peak, and decay boundaries.
Learning Objectives
- Implement an AdamW optimizer wired to a cosine learning-rate schedule with linear warmup.
- Compute the schedule's exact value at any step without floating-point drift across runs.
- Log gradient L2 norm side by side with the learning rate so training health is observable.
- Render the schedule to a text plot the eye can read and a CSV any tool can consume.
The Problem
The first thousand training updates are the loudest. The model's weights are still close to initialization. The optimizer's running second-moment estimate has not stabilised. The gradient norm is large and noisy. If the learning rate is at its peak during these updates the model either diverges outright or settles into a loss plateau it never escapes. The two well-known fixes are gradient clipping, which is the subject of Phase 19 lesson 45, and a learning-rate schedule that starts small and ramps up.
The cosine-with-warmup schedule has three regions. From step zero to step warmup_steps the learning rate scales linearly from zero to the configured peak lr_max. From step warmup_steps to step total_steps the learning rate follows the upper half of a cosine curve, decaying from lr_max to lr_min. After total_steps the learning rate is pinned at lr_min so a misconfigured trainer that overshoots does not silently exit the schedule.
The build problem is that schedules are easy to get wrong off by one. The off-by-one shows up six hours into a training run as a learning rate that is 1 percent too high or too low at the moment the model starts overfitting, which is invisible unless the schedule is exhaustively tested at boundaries.
The Concept
Warmup formula
For step in [0, warmup_steps] with warmup_steps > 0the learning rate is lr_max * step / warmup_steps. The degenerate warmup_steps = 0 case is treated as "no warmup": the schedule starts directly at lr_max at step zero and immediately enters cosine decay. Some test harnesses pass warmup_steps = 0 to check the schedule still produces a usable curve.
Cosine formula
For step in (warmup_steps, total_steps] the learning rate is lr_min + 0.5 * (lr_max - lr_min) * (1 + cos(pi * progress)) where progress = (step - warmup_steps) / max(1, total_steps - warmup_steps). At step = warmup_steps the cosine evaluates to cos(0) = 1which gives lr_maxmatching the warmup endpoint exactly. At step = total_steps the cosine evaluates to cos(pi) = -1which gives lr_minmatching the decay endpoint exactly.
The continuity at both endpoints is not an accident. It is the reason the schedule is implemented as a single function over stepnot as three different functions glued together. A glued schedule loses one boundary the first time lr_max is changed.
Floor after total steps
For step > total_steps the learning rate stays at lr_min. The contract is explicit: the schedule does not error out and does not extrapolate; it pins at the floor and lets the trainer log a warning. Trainers that need to extend training change the schedule's total_stepsnot the loop.
Gradient norm logging alongside the rate
The schedule is half of training health. The gradient norm is the other half. The training loop logs both per step. A divergent training run shows the gradient norm spike before the loss does; a well-tuned warmup keeps the norm rising linearly with the rate; a too-aggressive peak shows up as a norm that stays high after warmup. The dataset on disk is step, lr, grad_l2_norm, loss. The CSV is the only durable record.
Build It
code/main.py implements:
CosineWithWarmup- a stateless functionlr(step) -> floatover the configured schedule.TrainState- wraps a model, anAdamWoptimizer, and the schedule into a single step function.TrainState.step- runs one forward pass, one backward pass, logs gradient L2 norm, and applieslr(step)to the optimizer.plot_schedule_ascii- renders the schedule as a text plot the eye can read.write_schedule_csv- emits one row per step with the learning rate.
A demo at the bottom of the file builds a tiny nn.Linear model, trains for 20 steps over a fixed input batch, and prints the per-step learning rate, gradient norm, and loss. The schedule is also rendered as a text plot for the visual sanity check.
Run it:
python3 code/main.pyThe script exits zero and prints a per-step training log plus the schedule plot.
Production Patterns
Four patterns elevate the schedule to a production artifact.
Schedule lives in a config, not in code. The trainer reads warmup_stepstotal_stepslr_maxlr_min from a YAML or JSON config that is committed to git. The schedule is reproducible because the config is content-addressed; the schedule is auditable because the config is part of the PR diff.
Step counter is monotonic and decoupled from epochs. Some frameworks confuse step and epoch when the dataset is sharded or the dataloader restarts. The schedule reads global_step from the trainer's checkpoint, not from a local counter. A resumed run continues at the right schedule position because the step counter is the durable axis.
Schedule plot in the run directory. Every training run writes outputs/lr_schedule.png (or in this lesson a text plot) into its run directory. A reviewer who skims the directory can sanity-check the schedule without re-running anything. This catches the misconfigured-schedule class of bugs at PR time.
Log row schema is fixed. step, lr, grad_l2_norm, loss in that order. A downstream notebook or dashboard reads the schema; renaming a column without bumping a version invalidates every existing dashboard.
Use It
Production patterns:
- Sweep peak before sweeping anything else.
lr_maxis the most sensitive knob. Sweep it on a small model first; the optimallr_maxscales weakly with model size, so the small-model sweep is a strong prior. - Warmup is a fraction of total steps, not an absolute count. A 200-million-step run with 2,000 warmup steps starts at peak almost immediately; a 20,000-step run with the same number warms up for 10 percent. Configure warmup as a fraction (typical: 1-3 percent) so the schedule scales with training duration.
lr_minis non-zero on purpose. A floor that is 10 percent oflr_maxkeeps the optimizer learning during the long tail. Alr_min = 0schedule produces a training curve that looks great on a plot and a model that has not actually finished training.
Ship It
outputs/skill-cosine-warmup.md would, on a real project, describe which config carries the schedule, which trainer step the global counter is read from, and what lr_max sweep produced the deployed value. This lesson ships the engine.
Exercises
- Add an inverse-square-root variant of the schedule and compare it on a 200-step toy training run. Which curve produces the lower final loss?
- Add a
--restartflag that adds a second warmup attotal_steps / 2. Defend whether warm restarts improve or hurt on the toy run. - Add a unit test that the schedule is continuous: for every step in
[0, total_steps]the difference|lr(step+1) - lr(step)|is bounded bylr_max / warmup_steps. - Wire the schedule into a
torch.optim.lr_scheduler.LambdaLRso it composes with framework code. The lesson uses a plain step function; what does the wrapper change? - Add a
--plot-pngflag that writes a real plot viamatplotlib. Defend whether the lesson's text plot or the PNG is the better default for CI runs.
Key Terms
| Term | What people say | What it actually means |
|---|---|---|
| Warmup | "Slow start" | Linear ramp from zero to lr_max over the first warmup_steps updates |
| Cosine decay | "Smooth drop" | Upper-half cosine curve from lr_max to lr_min over the remaining steps |
| Floor | "After training" | The fixed lr_min value the schedule pins at past total_steps |
| Gradient norm | "L2 of grads" | The Euclidean norm of the concatenated gradient vector, logged each step |
| Global step | "Schedule axis" | A monotonic step counter that survives restarts and drives the schedule |
Further Reading
- Loshchilov and Hutter, SGDR: Stochastic Gradient Descent with Warm Restarts (arXiv 1608.03983) - the cosine schedule's reference paper
- Loshchilov and Hutter, Decoupled Weight Decay Regularization (arXiv 1711.05101) - AdamW's reference paper
- PyTorch torch.optim.lr_scheduler - how step functions compose with framework schedulers
- Phase 19 · 42 - the downloader whose corpus this schedule consumes
- Phase 19 · 43 - the dataloader the schedule co-evolves with
- Phase 19 · 45 - gradient clipping and AMP, the next layer in the loop
Check what you learned
6 questions. Answer each one to see why.
Why does the cosine-with-warmup schedule start at zero rather than at lr_max?
The full code
Every file for this lesson. Copy it, run it, then change it.
"""AdamW with cosine learning-rate schedule and linear warmup.
Implements:
- CosineWithWarmup, a stateless schedule whose lr(step) honors warmup, peak,
and decay boundaries exactly.
- TrainState, which wires an AdamW optimizer to the schedule and runs one
training step at a time, logging the learning rate and gradient L2 norm.
- plot_schedule_ascii and write_schedule_csv, deterministic helpers that
produce a text plot and a CSV the rest of the pipeline can read.
The demo at the bottom builds a tiny torch.nn.Linear model, trains for 20
steps on a fixed batch, prints a per-step log, and renders the schedule.
Run: python3 code/main.py
"""
from __future__ import annotations
import csv
import dataclasses
import math
import sys
from dataclasses import dataclass, field
from pathlib import Path
from typing import Callable, Iterable
try:
import torch
from torch import nn
except ImportError as exc:
raise SystemExit(
"torch is required for this lesson. Install with: pip install torch"
) from exc
PLOT_HEIGHT = 12
PLOT_WIDTH = 60
@dataclass
class CosineWithWarmup:
"""Stateless cosine-with-warmup learning-rate schedule.
Step indexing convention: step zero is the very first training update. At
step zero the rate is exactly zero (the warmup ramp starts there). At
step warmup_steps the rate is exactly lr_max. At step total_steps the
rate is exactly lr_min. Past total_steps the rate stays at lr_min.
"""
warmup_steps: int
total_steps: int
lr_max: float
lr_min: float = 0.0
def __post_init__(self) -> None:
if self.warmup_steps < 0:
raise ValueError("warmup_steps must be non-negative")
if self.total_steps <= 0:
raise ValueError("total_steps must be positive")
if self.warmup_steps >= self.total_steps:
raise ValueError("warmup_steps must be less than total_steps")
if self.lr_max <= 0:
raise ValueError("lr_max must be positive")
if self.lr_min < 0:
raise ValueError("lr_min must be non-negative")
if self.lr_min > self.lr_max:
raise ValueError("lr_min must not exceed lr_max")
def lr(self, step: int) -> float:
if step < 0:
raise ValueError(f"step must be non-negative, got {step}")
if self.warmup_steps > 0 and step <= self.warmup_steps:
return self.lr_max * (step / self.warmup_steps)
if step >= self.total_steps:
return self.lr_min
decay_span = max(1, self.total_steps - self.warmup_steps)
progress = (step - self.warmup_steps) / decay_span
cosine = 0.5 * (1.0 + math.cos(math.pi * progress))
return self.lr_min + (self.lr_max - self.lr_min) * cosine
def points(self, num_steps: int | None = None) -> list[tuple[int, float]]:
upper = self.total_steps if num_steps is None else num_steps
if upper <= 0:
return []
return [(step, self.lr(step)) for step in range(upper + 1)]
@dataclass
class StepLog:
"""One row of the per-step training log."""
step: int
lr: float
grad_l2_norm: float
loss: float
def to_csv_row(self) -> list[str]:
return [
str(self.step),
f"{self.lr:.10f}",
f"{self.grad_l2_norm:.10f}",
f"{self.loss:.10f}",
]
def gradient_l2_norm(parameters: Iterable[torch.nn.Parameter]) -> float:
"""Return the L2 norm of the concatenated gradient vector.
Mirrors `torch.nn.utils.get_total_norm` for the gradient case so the lesson
does not depend on a particular PyTorch version that may or may not expose
that helper.
"""
squared_sum = 0.0
for param in parameters:
if param.grad is None:
continue
grad = param.grad.detach()
squared_sum += float(grad.pow(2).sum().item())
return math.sqrt(squared_sum)
class TrainState:
"""Bind a model, an AdamW optimizer, a schedule, and a loss function.
The class owns the step counter so the schedule axis is the durable one.
"""
def __init__(
self,
model: nn.Module,
schedule: CosineWithWarmup,
loss_fn: Callable[[torch.Tensor, torch.Tensor], torch.Tensor],
weight_decay: float = 0.01,
betas: tuple[float, float] = (0.9, 0.95),
eps: float = 1e-8,
) -> None:
self.model = model
self.schedule = schedule
self.loss_fn = loss_fn
self.optimizer = torch.optim.AdamW(
model.parameters(),
lr=schedule.lr(0),
betas=betas,
eps=eps,
weight_decay=weight_decay,
)
self.global_step = 0
self._log: list[StepLog] = []
def set_lr(self, lr: float) -> None:
for group in self.optimizer.param_groups:
group["lr"] = lr
@property
def log(self) -> list[StepLog]:
return list(self._log)
def step(self, batch_inputs: torch.Tensor, batch_targets: torch.Tensor) -> StepLog:
self.model.train()
self.optimizer.zero_grad(set_to_none=True)
predictions = self.model(batch_inputs)
loss = self.loss_fn(predictions, batch_targets)
loss.backward()
grad_norm = gradient_l2_norm(self.model.parameters())
rate = self.schedule.lr(self.global_step)
self.set_lr(rate)
self.optimizer.step()
record = StepLog(
step=self.global_step,
lr=rate,
grad_l2_norm=grad_norm,
loss=float(loss.detach().item()),
)
self._log.append(record)
self.global_step += 1
return record
def plot_schedule_ascii(
schedule: CosineWithWarmup,
width: int = PLOT_WIDTH,
height: int = PLOT_HEIGHT,
) -> str:
"""Return a text plot of the schedule across [0, total_steps]."""
if width <= 2 or height <= 2:
raise ValueError("width and height must be at least 3")
total = schedule.total_steps
step_axis = [
int(round(i * total / max(1, width - 1))) for i in range(width)
]
rates = [schedule.lr(step) for step in step_axis]
upper = max(rates)
if upper <= 0:
upper = 1.0
grid = [[" "] * width for _ in range(height)]
for col, rate in enumerate(rates):
row = int(round((height - 1) * (1.0 - rate / upper)))
row = max(0, min(height - 1, row))
grid[row][col] = "*"
rows: list[str] = []
for r, row in enumerate(grid):
label = upper * (1.0 - r / max(1, height - 1))
rows.append(f"{label:8.6f} | " + "".join(row))
axis = " " * 10 + "+" + "-" * width
last_label = (
" " * 11
+ f"step 0"
+ " " * (width - len("step 0") - len(f"step {total}"))
+ f"step {total}"
)
return "\n".join(rows + [axis, last_label])
def write_schedule_csv(schedule: CosineWithWarmup, path: Path) -> None:
"""Write one row per step to a CSV with columns (step, lr)."""
path = Path(path)
path.parent.mkdir(parents=True, exist_ok=True)
with path.open("w", encoding="utf-8", newline="") as fh:
writer = csv.writer(fh)
writer.writerow(["step", "lr"])
for step, rate in schedule.points():
writer.writerow([step, f"{rate:.10f}"])
def write_step_log_csv(log: Iterable[StepLog], path: Path) -> None:
"""Write the training log to a CSV with the canonical schema."""
path = Path(path)
path.parent.mkdir(parents=True, exist_ok=True)
with path.open("w", encoding="utf-8", newline="") as fh:
writer = csv.writer(fh)
writer.writerow(["step", "lr", "grad_l2_norm", "loss"])
for row in log:
writer.writerow(row.to_csv_row())
@dataclass
class LinearWarmupConstant:
"""Alternative schedule: linear warmup followed by a flat lr_max plateau.
Useful as a baseline for ablations against the cosine variant. The same
contract: lr(0) is zero (with non-zero warmup) and the rate stays at
lr_max for steps beyond warmup_steps.
"""
warmup_steps: int
lr_max: float
def __post_init__(self) -> None:
if self.warmup_steps < 0:
raise ValueError("warmup_steps must be non-negative")
if self.lr_max <= 0:
raise ValueError("lr_max must be positive")
def lr(self, step: int) -> float:
if step < 0:
raise ValueError(f"step must be non-negative, got {step}")
if self.warmup_steps == 0:
return self.lr_max
if step >= self.warmup_steps:
return self.lr_max
return self.lr_max * (step / self.warmup_steps)
@dataclass
class InverseSqrtWarmup:
"""Linear warmup followed by an inverse-square-root decay.
Decays as lr_max * sqrt(warmup_steps / step) for step > warmup_steps. Used
historically in transformer training and useful as a comparison baseline.
"""
warmup_steps: int
lr_max: float
def __post_init__(self) -> None:
if self.warmup_steps <= 0:
raise ValueError("inverse-sqrt warmup requires warmup_steps > 0")
if self.lr_max <= 0:
raise ValueError("lr_max must be positive")
def lr(self, step: int) -> float:
if step < 0:
raise ValueError(f"step must be non-negative, got {step}")
if step <= self.warmup_steps:
return self.lr_max * (step / self.warmup_steps)
return self.lr_max * math.sqrt(self.warmup_steps / step)
@dataclass
class EWMA:
"""Exponentially weighted moving average of a scalar, useful for grad-norm smoothing."""
beta: float
value: float = 0.0
initialized: bool = False
def __post_init__(self) -> None:
if not 0 < self.beta < 1:
raise ValueError("beta must be in (0, 1)")
def update(self, sample: float) -> float:
if not self.initialized:
self.value = float(sample)
self.initialized = True
return self.value
self.value = self.beta * self.value + (1.0 - self.beta) * float(sample)
return self.value
@dataclass
class StepLogSummary:
"""Reduction of a per-step log to the numbers a reviewer scans first."""
steps: int
lr_peak: float
lr_final: float
grad_l2_peak: float
loss_initial: float
loss_final: float
loss_delta: float
def summarize_step_log(log: Iterable[StepLog]) -> StepLogSummary:
rows = list(log)
if not rows:
raise ValueError("step log is empty")
return StepLogSummary(
steps=len(rows),
lr_peak=max(row.lr for row in rows),
lr_final=rows[-1].lr,
grad_l2_peak=max(row.grad_l2_norm for row in rows),
loss_initial=rows[0].loss,
loss_final=rows[-1].loss,
loss_delta=rows[-1].loss - rows[0].loss,
)
def split_decay_groups(
model: nn.Module,
weight_decay: float = 0.01,
no_decay_names: tuple[str, ...] = ("bias", "LayerNorm.weight", "layer_norm.weight"),
) -> list[dict[str, object]]:
"""Split model parameters into a decay and a no-decay group.
The convention for transformer training is to apply weight decay to dense
weight matrices but not to biases or LayerNorm gain parameters. This helper
returns the two parameter-group dicts AdamW accepts.
"""
decay_params: list[nn.Parameter] = []
no_decay_params: list[nn.Parameter] = []
for name, param in model.named_parameters():
if not param.requires_grad:
continue
if any(needle in name for needle in no_decay_names):
no_decay_params.append(param)
else:
decay_params.append(param)
groups: list[dict[str, object]] = []
if decay_params:
groups.append({"params": decay_params, "weight_decay": weight_decay})
if no_decay_params:
groups.append({"params": no_decay_params, "weight_decay": 0.0})
return groups
def build_toy_model(
in_dim: int = 16,
out_dim: int = 4,
seed: int = 7,
) -> tuple[nn.Module, torch.Tensor, torch.Tensor]:
"""Tiny linear model with a fixed batch for the demo."""
torch.manual_seed(seed)
model = nn.Sequential(nn.Linear(in_dim, 32), nn.GELU(), nn.Linear(32, out_dim))
inputs = torch.randn(8, in_dim)
targets = torch.randn(8, out_dim)
return model, inputs, targets
def run_demo() -> int:
"""Run 20 training steps on a toy model and render the schedule."""
model, inputs, targets = build_toy_model()
schedule = CosineWithWarmup(
warmup_steps=4,
total_steps=20,
lr_max=1e-2,
lr_min=1e-4,
)
state = TrainState(
model=model,
schedule=schedule,
loss_fn=nn.functional.mse_loss,
)
for _ in range(20):
record = state.step(inputs, targets)
print(
f"step={record.step:>3} lr={record.lr:.6f} "
f"grad_l2={record.grad_l2_norm:.6f} loss={record.loss:.6f}"
)
print()
print("learning rate schedule:")
print(plot_schedule_ascii(schedule, width=40, height=10))
summary = summarize_step_log(state.log)
print()
print(
f"summary: steps={summary.steps} lr_peak={summary.lr_peak:.6f} "
f"grad_l2_peak={summary.grad_l2_peak:.6f} loss_delta={summary.loss_delta:.6f}"
)
return 0
if __name__ == "__main__":
sys.exit(run_demo())