Numerical Stability
Floating point is a leaky abstraction. It will bite you during training, and you will not see it coming.
120 minutesBuildPythonMembers
Learning Objectives
- Implement numerically stable softmax and log-sum-exp using the max-subtraction trick
- Identify overflow, underflow, and catastrophic cancellation in floating-point computations
- Verify analytical gradients against numerical gradients using centered finite differences
- Explain why bfloat16 is preferred over float16 for training and how loss scaling prevents gradient underflow