Data Parallel DDP From Scratch
DistributedDataParallel is a hook on top of allreduce. Wrap a model, broadcast the initial parameters from rank 0 so every rank starts identical, install a backward hook on every parameter that issues an allreduce of the gradient, and the rest is gradient descent. The whole pattern is 200 lines.
90 minBuildPythonMembers
Learning Objectives
- Wire a
DistributedDataParallel-shaped wrapper that broadcasts initial parameters and allreduces gradients after backward. - Spawn N CPU ranks with
torch.multiprocessing.spawnover the gloo backend with file-based rendezvous. - Prove gradient-sync correctness by training the same model on the same data sequentially and showing per-step parameter equivalence.
- Defend the use of buckets (gradient fusion) and overlap (comm during backward) as the two changes that turn a working DDP into a production DDP.