Get Started

End-to-End Distributed Training

Lessons 76 through 80 each built one piece. This is the assembly: a tiny GPT trained across 4 simulated ranks with DDP for gradient sync, ZeRO-1 for optimiser-state sharding, and a sharded checkpoint at the halfway mark. The demo runs 20 steps, self-terminates, prints a loss curve plus a memory profile, and writes a resumable checkpoint.

90 minBuildPythonMembers

Learning Objectives

  • Compose DDP (lesson 77) plus ZeRO-1 (lesson 78) plus sharded checkpoints (lesson 80) into one training loop.
  • Train a 2-layer transformer language model on a small synthetic corpus for 20 steps across 4 simulated ranks.
  • Print a per-step loss table, a per-rank memory profile, and a checkpoint manifest that resumes byte-equal on the same world size.
  • Defend the composition: each piece is independently testable in earlier lessons and this lesson proves they compose.