Get Started

Checkpoint Save and Resume

Train interrupts kill runs; checkpoints let them continue. Save model, optimizer, scheduler, loss history, step counter, and RNG state, atomically, so a kill at any moment leaves a valid file on disk.

90 minutesBuildPythonMembers

Learning Objectives

  • Capture the full training state into a single payload that can be reloaded into a fresh process.
  • Implement atomic save with write-to-temp then rename so a crash never leaves a half-written file.
  • Restore the RNG state for Python, NumPy, and PyTorch so the post-resume loss matches the uninterrupted baseline.
  • Build a sharded checkpoint layout for models that no longer fit in a single file, with hash-verified shards and a JSON index.