Get Started

Numerical Stability

Floating point is a leaky abstraction. It will bite you during training, and you will not see it coming.

120 minutesBuildPythonMembers

Learning Objectives

  • Implement numerically stable softmax and log-sum-exp using the max-subtraction trick
  • Identify overflow, underflow, and catastrophic cancellation in floating-point computations
  • Verify analytical gradients against numerical gradients using centered finite differences
  • Explain why bfloat16 is preferred over float16 for training and how loss scaling prevents gradient underflow