Get Started

Cosine LR with Linear Warmup

The learning-rate schedule is the second most important decision after the loss function. AdamW with a cosine decay and a linear warmup is the modern default for language-model training because it lets the model see a small effective step size during the brittle first thousand updates, ramps up to a configured peak, and decays smoothly back toward zero. This lesson builds that schedule, plots the curve over training steps, logs gradient norms next to the schedule, and proves the schedule honors warmup, peak, and decay boundaries.

90 minutesBuildPythonFree lesson

Learning Objectives

  • Implement an AdamW optimizer wired to a cosine learning-rate schedule with linear warmup.
  • Compute the schedule's exact value at any step without floating-point drift across runs.
  • Log gradient L2 norm side by side with the learning rate so training health is observable.
  • Render the schedule to a text plot the eye can read and a CSV any tool can consume.

The Problem

The first thousand training updates are the loudest. The model's weights are still close to initialization. The optimizer's running second-moment estimate has not stabilised. The gradient norm is large and noisy. If the learning rate is at its peak during these updates the model either diverges outright or settles into a loss plateau it never escapes. The two well-known fixes are gradient clipping, which is the subject of Phase 19 lesson 45, and a learning-rate schedule that starts small and ramps up.

The cosine-with-warmup schedule has three regions. From step zero to step warmup_steps the learning rate scales linearly from zero to the configured peak lr_max. From step warmup_steps to step total_steps the learning rate follows the upper half of a cosine curve, decaying from lr_max to lr_min. After total_steps the learning rate is pinned at lr_min so a misconfigured trainer that overshoots does not silently exit the schedule.

The build problem is that schedules are easy to get wrong off by one. The off-by-one shows up six hours into a training run as a learning rate that is 1 percent too high or too low at the moment the model starts overfitting, which is invisible unless the schedule is exhaustively tested at boundaries.

The Concept

Warmup formula

For step in [0, warmup_steps] with warmup_steps > 0the learning rate is lr_max * step / warmup_steps. The degenerate warmup_steps = 0 case is treated as "no warmup": the schedule starts directly at lr_max at step zero and immediately enters cosine decay. Some test harnesses pass warmup_steps = 0 to check the schedule still produces a usable curve.

Cosine formula

For step in (warmup_steps, total_steps] the learning rate is lr_min + 0.5 * (lr_max - lr_min) * (1 + cos(pi * progress)) where progress = (step - warmup_steps) / max(1, total_steps - warmup_steps). At step = warmup_steps the cosine evaluates to cos(0) = 1which gives lr_maxmatching the warmup endpoint exactly. At step = total_steps the cosine evaluates to cos(pi) = -1which gives lr_minmatching the decay endpoint exactly.

The continuity at both endpoints is not an accident. It is the reason the schedule is implemented as a single function over stepnot as three different functions glued together. A glued schedule loses one boundary the first time lr_max is changed.

Floor after total steps

For step > total_steps the learning rate stays at lr_min. The contract is explicit: the schedule does not error out and does not extrapolate; it pins at the floor and lets the trainer log a warning. Trainers that need to extend training change the schedule's total_stepsnot the loop.

Gradient norm logging alongside the rate

The schedule is half of training health. The gradient norm is the other half. The training loop logs both per step. A divergent training run shows the gradient norm spike before the loss does; a well-tuned warmup keeps the norm rising linearly with the rate; a too-aggressive peak shows up as a norm that stays high after warmup. The dataset on disk is step, lr, grad_l2_norm, loss. The CSV is the only durable record.

Build It

code/main.py implements:

  • CosineWithWarmup - a stateless function lr(step) -> float over the configured schedule.
  • TrainState - wraps a model, an AdamW optimizer, and the schedule into a single step function.
  • TrainState.step - runs one forward pass, one backward pass, logs gradient L2 norm, and applies lr(step) to the optimizer.
  • plot_schedule_ascii - renders the schedule as a text plot the eye can read.
  • write_schedule_csv - emits one row per step with the learning rate.

A demo at the bottom of the file builds a tiny nn.Linear model, trains for 20 steps over a fixed input batch, and prints the per-step learning rate, gradient norm, and loss. The schedule is also rendered as a text plot for the visual sanity check.

Run it:

Terminal
python3 code/main.py

The script exits zero and prints a per-step training log plus the schedule plot.

Production Patterns

Four patterns elevate the schedule to a production artifact.

Schedule lives in a config, not in code. The trainer reads warmup_stepstotal_stepslr_maxlr_min from a YAML or JSON config that is committed to git. The schedule is reproducible because the config is content-addressed; the schedule is auditable because the config is part of the PR diff.

Step counter is monotonic and decoupled from epochs. Some frameworks confuse step and epoch when the dataset is sharded or the dataloader restarts. The schedule reads global_step from the trainer's checkpoint, not from a local counter. A resumed run continues at the right schedule position because the step counter is the durable axis.

Schedule plot in the run directory. Every training run writes outputs/lr_schedule.png (or in this lesson a text plot) into its run directory. A reviewer who skims the directory can sanity-check the schedule without re-running anything. This catches the misconfigured-schedule class of bugs at PR time.

Log row schema is fixed. step, lr, grad_l2_norm, loss in that order. A downstream notebook or dashboard reads the schema; renaming a column without bumping a version invalidates every existing dashboard.

Use It

Production patterns:

  • Sweep peak before sweeping anything else. lr_max is the most sensitive knob. Sweep it on a small model first; the optimal lr_max scales weakly with model size, so the small-model sweep is a strong prior.
  • Warmup is a fraction of total steps, not an absolute count. A 200-million-step run with 2,000 warmup steps starts at peak almost immediately; a 20,000-step run with the same number warms up for 10 percent. Configure warmup as a fraction (typical: 1-3 percent) so the schedule scales with training duration.
  • lr_min is non-zero on purpose. A floor that is 10 percent of lr_max keeps the optimizer learning during the long tail. A lr_min = 0 schedule produces a training curve that looks great on a plot and a model that has not actually finished training.

Ship It

outputs/skill-cosine-warmup.md would, on a real project, describe which config carries the schedule, which trainer step the global counter is read from, and what lr_max sweep produced the deployed value. This lesson ships the engine.

Exercises

  1. Add an inverse-square-root variant of the schedule and compare it on a 200-step toy training run. Which curve produces the lower final loss?
  2. Add a --restart flag that adds a second warmup at total_steps / 2. Defend whether warm restarts improve or hurt on the toy run.
  3. Add a unit test that the schedule is continuous: for every step in [0, total_steps] the difference |lr(step+1) - lr(step)| is bounded by lr_max / warmup_steps.
  4. Wire the schedule into a torch.optim.lr_scheduler.LambdaLR so it composes with framework code. The lesson uses a plain step function; what does the wrapper change?
  5. Add a --plot-png flag that writes a real plot via matplotlib. Defend whether the lesson's text plot or the PNG is the better default for CI runs.

Key Terms

TermWhat people sayWhat it actually means
Warmup"Slow start"Linear ramp from zero to lr_max over the first warmup_steps updates
Cosine decay"Smooth drop"Upper-half cosine curve from lr_max to lr_min over the remaining steps
Floor"After training"The fixed lr_min value the schedule pins at past total_steps
Gradient norm"L2 of grads"The Euclidean norm of the concatenated gradient vector, logged each step
Global step"Schedule axis"A monotonic step counter that survives restarts and drives the schedule

Further Reading

Check what you learned

6 questions. Answer each one to see why.

Question 1 of 60 correct

Why does the cosine-with-warmup schedule start at zero rather than at lr_max?

The full code

Every file for this lesson. Copy it, run it, then change it.

main.py
"""AdamW with cosine learning-rate schedule and linear warmup.

Implements:
- CosineWithWarmup, a stateless schedule whose lr(step) honors warmup, peak,
  and decay boundaries exactly.
- TrainState, which wires an AdamW optimizer to the schedule and runs one
  training step at a time, logging the learning rate and gradient L2 norm.
- plot_schedule_ascii and write_schedule_csv, deterministic helpers that
  produce a text plot and a CSV the rest of the pipeline can read.

The demo at the bottom builds a tiny torch.nn.Linear model, trains for 20
steps on a fixed batch, prints a per-step log, and renders the schedule.
Run: python3 code/main.py
"""

from __future__ import annotations

import csv
import dataclasses
import math
import sys
from dataclasses import dataclass, field
from pathlib import Path
from typing import Callable, Iterable

try:
    import torch
    from torch import nn
except ImportError as exc:
    raise SystemExit(
        "torch is required for this lesson. Install with: pip install torch"
    ) from exc


PLOT_HEIGHT = 12
PLOT_WIDTH = 60


@dataclass
class CosineWithWarmup:
    """Stateless cosine-with-warmup learning-rate schedule.

    Step indexing convention: step zero is the very first training update. At
    step zero the rate is exactly zero (the warmup ramp starts there). At
    step warmup_steps the rate is exactly lr_max. At step total_steps the
    rate is exactly lr_min. Past total_steps the rate stays at lr_min.
    """

    warmup_steps: int
    total_steps: int
    lr_max: float
    lr_min: float = 0.0

    def __post_init__(self) -> None:
        if self.warmup_steps < 0:
            raise ValueError("warmup_steps must be non-negative")
        if self.total_steps <= 0:
            raise ValueError("total_steps must be positive")
        if self.warmup_steps >= self.total_steps:
            raise ValueError("warmup_steps must be less than total_steps")
        if self.lr_max <= 0:
            raise ValueError("lr_max must be positive")
        if self.lr_min < 0:
            raise ValueError("lr_min must be non-negative")
        if self.lr_min > self.lr_max:
            raise ValueError("lr_min must not exceed lr_max")

    def lr(self, step: int) -> float:
        if step < 0:
            raise ValueError(f"step must be non-negative, got {step}")
        if self.warmup_steps > 0 and step <= self.warmup_steps:
            return self.lr_max * (step / self.warmup_steps)
        if step >= self.total_steps:
            return self.lr_min
        decay_span = max(1, self.total_steps - self.warmup_steps)
        progress = (step - self.warmup_steps) / decay_span
        cosine = 0.5 * (1.0 + math.cos(math.pi * progress))
        return self.lr_min + (self.lr_max - self.lr_min) * cosine

    def points(self, num_steps: int | None = None) -> list[tuple[int, float]]:
        upper = self.total_steps if num_steps is None else num_steps
        if upper <= 0:
            return []
        return [(step, self.lr(step)) for step in range(upper + 1)]


@dataclass
class StepLog:
    """One row of the per-step training log."""

    step: int
    lr: float
    grad_l2_norm: float
    loss: float

    def to_csv_row(self) -> list[str]:
        return [
            str(self.step),
            f"{self.lr:.10f}",
            f"{self.grad_l2_norm:.10f}",
            f"{self.loss:.10f}",
        ]


def gradient_l2_norm(parameters: Iterable[torch.nn.Parameter]) -> float:
    """Return the L2 norm of the concatenated gradient vector.

    Mirrors `torch.nn.utils.get_total_norm` for the gradient case so the lesson
    does not depend on a particular PyTorch version that may or may not expose
    that helper.
    """

    squared_sum = 0.0
    for param in parameters:
        if param.grad is None:
            continue
        grad = param.grad.detach()
        squared_sum += float(grad.pow(2).sum().item())
    return math.sqrt(squared_sum)


class TrainState:
    """Bind a model, an AdamW optimizer, a schedule, and a loss function.

    The class owns the step counter so the schedule axis is the durable one.
    """

    def __init__(
        self,
        model: nn.Module,
        schedule: CosineWithWarmup,
        loss_fn: Callable[[torch.Tensor, torch.Tensor], torch.Tensor],
        weight_decay: float = 0.01,
        betas: tuple[float, float] = (0.9, 0.95),
        eps: float = 1e-8,
    ) -> None:
        self.model = model
        self.schedule = schedule
        self.loss_fn = loss_fn
        self.optimizer = torch.optim.AdamW(
            model.parameters(),
            lr=schedule.lr(0),
            betas=betas,
            eps=eps,
            weight_decay=weight_decay,
        )
        self.global_step = 0
        self._log: list[StepLog] = []

    def set_lr(self, lr: float) -> None:
        for group in self.optimizer.param_groups:
            group["lr"] = lr

    @property
    def log(self) -> list[StepLog]:
        return list(self._log)

    def step(self, batch_inputs: torch.Tensor, batch_targets: torch.Tensor) -> StepLog:
        self.model.train()
        self.optimizer.zero_grad(set_to_none=True)
        predictions = self.model(batch_inputs)
        loss = self.loss_fn(predictions, batch_targets)
        loss.backward()
        grad_norm = gradient_l2_norm(self.model.parameters())
        rate = self.schedule.lr(self.global_step)
        self.set_lr(rate)
        self.optimizer.step()
        record = StepLog(
            step=self.global_step,
            lr=rate,
            grad_l2_norm=grad_norm,
            loss=float(loss.detach().item()),
        )
        self._log.append(record)
        self.global_step += 1
        return record


def plot_schedule_ascii(
    schedule: CosineWithWarmup,
    width: int = PLOT_WIDTH,
    height: int = PLOT_HEIGHT,
) -> str:
    """Return a text plot of the schedule across [0, total_steps]."""

    if width <= 2 or height <= 2:
        raise ValueError("width and height must be at least 3")
    total = schedule.total_steps
    step_axis = [
        int(round(i * total / max(1, width - 1))) for i in range(width)
    ]
    rates = [schedule.lr(step) for step in step_axis]
    upper = max(rates)
    if upper <= 0:
        upper = 1.0

    grid = [[" "] * width for _ in range(height)]
    for col, rate in enumerate(rates):
        row = int(round((height - 1) * (1.0 - rate / upper)))
        row = max(0, min(height - 1, row))
        grid[row][col] = "*"

    rows: list[str] = []
    for r, row in enumerate(grid):
        label = upper * (1.0 - r / max(1, height - 1))
        rows.append(f"{label:8.6f} | " + "".join(row))
    axis = " " * 10 + "+" + "-" * width
    last_label = (
        " " * 11
        + f"step 0"
        + " " * (width - len("step 0") - len(f"step {total}"))
        + f"step {total}"
    )
    return "\n".join(rows + [axis, last_label])


def write_schedule_csv(schedule: CosineWithWarmup, path: Path) -> None:
    """Write one row per step to a CSV with columns (step, lr)."""

    path = Path(path)
    path.parent.mkdir(parents=True, exist_ok=True)
    with path.open("w", encoding="utf-8", newline="") as fh:
        writer = csv.writer(fh)
        writer.writerow(["step", "lr"])
        for step, rate in schedule.points():
            writer.writerow([step, f"{rate:.10f}"])


def write_step_log_csv(log: Iterable[StepLog], path: Path) -> None:
    """Write the training log to a CSV with the canonical schema."""

    path = Path(path)
    path.parent.mkdir(parents=True, exist_ok=True)
    with path.open("w", encoding="utf-8", newline="") as fh:
        writer = csv.writer(fh)
        writer.writerow(["step", "lr", "grad_l2_norm", "loss"])
        for row in log:
            writer.writerow(row.to_csv_row())


@dataclass
class LinearWarmupConstant:
    """Alternative schedule: linear warmup followed by a flat lr_max plateau.

    Useful as a baseline for ablations against the cosine variant. The same
    contract: lr(0) is zero (with non-zero warmup) and the rate stays at
    lr_max for steps beyond warmup_steps.
    """

    warmup_steps: int
    lr_max: float

    def __post_init__(self) -> None:
        if self.warmup_steps < 0:
            raise ValueError("warmup_steps must be non-negative")
        if self.lr_max <= 0:
            raise ValueError("lr_max must be positive")

    def lr(self, step: int) -> float:
        if step < 0:
            raise ValueError(f"step must be non-negative, got {step}")
        if self.warmup_steps == 0:
            return self.lr_max
        if step >= self.warmup_steps:
            return self.lr_max
        return self.lr_max * (step / self.warmup_steps)


@dataclass
class InverseSqrtWarmup:
    """Linear warmup followed by an inverse-square-root decay.

    Decays as lr_max * sqrt(warmup_steps / step) for step > warmup_steps. Used
    historically in transformer training and useful as a comparison baseline.
    """

    warmup_steps: int
    lr_max: float

    def __post_init__(self) -> None:
        if self.warmup_steps <= 0:
            raise ValueError("inverse-sqrt warmup requires warmup_steps > 0")
        if self.lr_max <= 0:
            raise ValueError("lr_max must be positive")

    def lr(self, step: int) -> float:
        if step < 0:
            raise ValueError(f"step must be non-negative, got {step}")
        if step <= self.warmup_steps:
            return self.lr_max * (step / self.warmup_steps)
        return self.lr_max * math.sqrt(self.warmup_steps / step)


@dataclass
class EWMA:
    """Exponentially weighted moving average of a scalar, useful for grad-norm smoothing."""

    beta: float
    value: float = 0.0
    initialized: bool = False

    def __post_init__(self) -> None:
        if not 0 < self.beta < 1:
            raise ValueError("beta must be in (0, 1)")

    def update(self, sample: float) -> float:
        if not self.initialized:
            self.value = float(sample)
            self.initialized = True
            return self.value
        self.value = self.beta * self.value + (1.0 - self.beta) * float(sample)
        return self.value


@dataclass
class StepLogSummary:
    """Reduction of a per-step log to the numbers a reviewer scans first."""

    steps: int
    lr_peak: float
    lr_final: float
    grad_l2_peak: float
    loss_initial: float
    loss_final: float
    loss_delta: float


def summarize_step_log(log: Iterable[StepLog]) -> StepLogSummary:
    rows = list(log)
    if not rows:
        raise ValueError("step log is empty")
    return StepLogSummary(
        steps=len(rows),
        lr_peak=max(row.lr for row in rows),
        lr_final=rows[-1].lr,
        grad_l2_peak=max(row.grad_l2_norm for row in rows),
        loss_initial=rows[0].loss,
        loss_final=rows[-1].loss,
        loss_delta=rows[-1].loss - rows[0].loss,
    )


def split_decay_groups(
    model: nn.Module,
    weight_decay: float = 0.01,
    no_decay_names: tuple[str, ...] = ("bias", "LayerNorm.weight", "layer_norm.weight"),
) -> list[dict[str, object]]:
    """Split model parameters into a decay and a no-decay group.

    The convention for transformer training is to apply weight decay to dense
    weight matrices but not to biases or LayerNorm gain parameters. This helper
    returns the two parameter-group dicts AdamW accepts.
    """

    decay_params: list[nn.Parameter] = []
    no_decay_params: list[nn.Parameter] = []
    for name, param in model.named_parameters():
        if not param.requires_grad:
            continue
        if any(needle in name for needle in no_decay_names):
            no_decay_params.append(param)
        else:
            decay_params.append(param)
    groups: list[dict[str, object]] = []
    if decay_params:
        groups.append({"params": decay_params, "weight_decay": weight_decay})
    if no_decay_params:
        groups.append({"params": no_decay_params, "weight_decay": 0.0})
    return groups


def build_toy_model(
    in_dim: int = 16,
    out_dim: int = 4,
    seed: int = 7,
) -> tuple[nn.Module, torch.Tensor, torch.Tensor]:
    """Tiny linear model with a fixed batch for the demo."""

    torch.manual_seed(seed)
    model = nn.Sequential(nn.Linear(in_dim, 32), nn.GELU(), nn.Linear(32, out_dim))
    inputs = torch.randn(8, in_dim)
    targets = torch.randn(8, out_dim)
    return model, inputs, targets


def run_demo() -> int:
    """Run 20 training steps on a toy model and render the schedule."""

    model, inputs, targets = build_toy_model()
    schedule = CosineWithWarmup(
        warmup_steps=4,
        total_steps=20,
        lr_max=1e-2,
        lr_min=1e-4,
    )
    state = TrainState(
        model=model,
        schedule=schedule,
        loss_fn=nn.functional.mse_loss,
    )
    for _ in range(20):
        record = state.step(inputs, targets)
        print(
            f"step={record.step:>3} lr={record.lr:.6f} "
            f"grad_l2={record.grad_l2_norm:.6f} loss={record.loss:.6f}"
        )
    print()
    print("learning rate schedule:")
    print(plot_schedule_ascii(schedule, width=40, height=10))
    summary = summarize_step_log(state.log)
    print()
    print(
        f"summary: steps={summary.steps} lr_peak={summary.lr_peak:.6f} "
        f"grad_l2_peak={summary.grad_l2_peak:.6f} loss_delta={summary.loss_delta:.6f}"
    )
    return 0


if __name__ == "__main__":
    sys.exit(run_demo())