Collective Ops From Scratch
The four collective operations that hold distributed training together are allreduce, broadcast, allgather, and reduce_scatter. Every other primitive a training framework offers is a wrapper around these. Build them once over a multiprocessing.Queue mesh, verify them against a reference implementation, and the rest of the track becomes plumbing.
90 minBuildPythonMembers
Learning Objectives
- Implement ring allreduce in two passes (reduce-scatter then allgather) and prove the per-rank communication volume is 2(N-1)/N bytes per element.
- Build broadcast, allgather, and reduce_scatter on top of point-to-point sends over
multiprocessing.Queue. - Verify every primitive against a
torch.distributedgloo reference for the same input. - Defend the choice of ring versus tree on cluster shape, latency floor, and bandwidth ceiling.