Get Started

Gradient Accumulation

Train at an effective batch you cannot afford, one micro-batch at a time. Scale the loss, hold the optimizer step, and let the gradients pile up.

90 minutesBuildPythonMembers

Learning Objectives

  • Derive the effective batch identity: effective_batch = micro_batch * accum_steps.
  • Implement loss-per-micro-batch scaling so the accumulated gradient matches a single full-batch backward.
  • Skip optimizer synchronization until the last micro-batch (sync-on-last-step).
  • Read a throughput against effective batch curve and explain the diminishing return.