Probability and Distributions
Probability is the language AI uses to express uncertainty.
Learning Objectives
- Implement PMFs and PDFs from scratch for Bernoulli, categorical, Poisson, uniform, and normal distributions
- Compute expected value, variance, and use the Central Limit Theorem to explain why Gaussians dominate
- Build softmax and log-softmax functions with the numerical stability trick (subtract max logit)
- Calculate cross-entropy loss from logits and connect it to negative log-likelihood
The Problem
A classifier outputs [0.03, 0.91, 0.06]. A language model picks the next word from 50,000 candidates. A diffusion model generates images by sampling from learned distributions. All of these are probability in action.
Every prediction a model makes is a probability distribution. Every loss function measures how far the predicted distribution is from the true one. Every training step adjusts parameters to make one distribution look more like another. Without probability, you cannot read a single ML paper, debug a single model, or understand why your training loss is NaN.
The Concept
Events, Sample Spaces, and Probability
The sample space $S$ is the set of all possible outcomes. An event is a subset of the sample space. Probability maps events to numbers between 0 and 1.
Coin flip:
S = {H, T}
P(H) = 0.5, P(T) = 0.5
Single die roll:
S = {1, 2, 3, 4, 5, 6}
P(even) = P({2, 4, 6}) = 3/6 = 0.5Three axioms define all of probability:
- $P(A) \ge 0$ for any event $A$
- $P(S) = 1$ (something always happens)
- $P(A \text{ or } B) = P(A) + P(B)$ when $A$ and $B$ cannot both occur
Everything else (Bayes' theorem, expectations, distributions) follows from these three rules.
Conditional Probability and Independence
$P(A \mid B)$ is the probability of $A$ given that $B$ happened.
P(A|B) = P(A and B) / P(B)
Example: deck of cards
P(King | Face card) = P(King and Face card) / P(Face card)
= (4/52) / (12/52)
= 4/12 = 1/3Two events are independent when knowing one tells you nothing about the other:
Coin flips are independent. Drawing cards without replacement is not.
Probability Mass Functions vs Probability Density Functions
Discrete random variables have a probability mass function (PMF). Each outcome has a specific probability that you can read off directly.
PMF: P(X = k)
Fair die:
P(X = 1) = 1/6
P(X = 2) = 1/6
...
P(X = 6) = 1/6
Sum of all probabilities = 1Continuous random variables have a probability density function (PDF). The density at a single point is not a probability. Probability comes from integrating the density over an interval.
PDF: f(x)
P(a <= X <= b) = integral of f(x) from a to b
f(x) can be greater than 1 (density, not probability)
integral from -inf to +inf of f(x) dx = 1This distinction matters in ML. Classification outputs are PMFs (discrete choices). VAE latent spaces use PDFs (continuous).
Common Distributions
Bernoulli: one trial, two outcomes. Models binary classification.
Categorical: one trial, $k$ outcomes. Models multi-class classification (softmax output).
Uniform: all outcomes equally likely. Used for random initialization.
Normal (Gaussian): the bell curve. Parameterized by mean ($\mu$) and variance ($\sigma^2$).
f(x) = (1 / sqrt(2*pi*sigma^2)) * exp(-(x - mu)^2 / (2*sigma^2))
Standard normal: mu = 0, sigma = 1
68% of data within 1 sigma
95% within 2 sigma
99.7% within 3 sigmaPoisson: counts of rare events in a fixed interval. Models event rates.
Expected Value and Variance
Expected value is the weighted average outcome.
Variance measures spread around the mean.
In ML, expected value appears as the loss function (average loss over the data distribution). Variance tells you about model stability. High variance in gradients means noisy training.
Joint and Marginal Distributions
A joint distribution $P(X, Y)$ describes two random variables together.
Joint PMF example ($X$ = weather, $Y$ = umbrella):
| $Y=0$ (no umbrella) | $Y=1$ (umbrella) | Marginal $P(X)$ | |
|---|---|---|---|
| $X=0$ (sun) | 0.40 | 0.10 | $P(X=0) = 0.50$ |
| $X=1$ (rain) | 0.05 | 0.45 | $P(X=1) = 0.50$ |
| Marginal $P(Y)$ | $P(Y=0) = 0.45$ | $P(Y=1) = 0.55$ | 1.00 |
The marginal distribution sums out the other variable:
The row and column totals in the table above are the marginals.
Why the Normal Distribution Shows Up Everywhere
The Central Limit Theorem: the sum (or average) of many independent random variables converges to a normal distribution, regardless of the original distribution.
Roll 1 die: uniform distribution (flat)
Average of 2 dice: triangular (peaked)
Average of 30 dice: nearly perfect bell curve
This works for ANY starting distribution.This is why:
- Measurement errors are approximately normal (many small independent sources)
- Weight initializations in neural networks use normal distributions
- Gradient noise in SGD is approximately normal (sum of many sample gradients)
- The normal distribution is the maximum entropy distribution for a given mean and variance
Log Probabilities
Raw probabilities cause numerical problems. Multiplying many small probabilities together quickly underflows to zero.
Log probabilities fix this. Multiplications become additions.
Rules:
- $\log(a \cdot b) = \log(a) + \log(b)$
- log probabilities are always $\le 0$ (since $0 < P \le 1$)
- More negative = less likely
- Cross-entropy loss is the negative log probability of the correct class
Softmax as a Probability Distribution
Neural networks output raw scores (logits). Softmax converts them into a valid probability distribution.
softmax(z_i) = exp(z_i) / sum(exp(z_j) for all j)
Properties:
- All outputs are in (0, 1)
- All outputs sum to 1
- Preserves relative ordering of inputs
- exp() amplifies differences between logitsThe softmax trick: subtract the max logit before exponentiating to prevent overflow.
z = [100, 101, 102]
exp(102) = overflow
z_shifted = z - max(z) = [-2, -1, 0]
exp(0) = 1 (safe)
Same result, no overflow.Log-softmax combines softmax and log for numerical stability. PyTorch uses this internally for cross-entropy loss.
Sampling
Sampling means drawing random values from a distribution. In ML:
- Dropout randomly samples which neurons to zero out
- Data augmentation samples random transformations
- Language models sample the next token from the predicted distribution
- Diffusion models sample noise and progressively denoise
Sampling from arbitrary distributions requires techniques like inverse transform sampling, rejection sampling, or the reparameterization trick (used in VAEs).
Build It
Step 1: Probability basics
import math
import random
def factorial(n):
result = 1
for i in range(2, n + 1):
result *= i
return result
def combinations(n, k):
return factorial(n) // (factorial(k) * factorial(n - k))
def conditional_probability(p_a_and_b, p_b):
return p_a_and_b / p_b
p_king_given_face = conditional_probability(4/52, 12/52)
print(f"P(King | Face card) = {p_king_given_face:.4f}")Step 2: PMF and PDF from scratch
def bernoulli_pmf(k, p):
return p if k == 1 else (1 - p)
def categorical_pmf(k, probs):
return probs[k]
def poisson_pmf(k, lam):
return (lam ** k) * math.exp(-lam) / factorial(k)
def uniform_pdf(x, a, b):
if a <= x <= b:
return 1.0 / (b - a)
return 0.0
def normal_pdf(x, mu, sigma):
coeff = 1.0 / (sigma * math.sqrt(2 * math.pi))
exponent = -0.5 * ((x - mu) / sigma) ** 2
return coeff * math.exp(exponent)Step 3: Expected value and variance
def expected_value(values, probabilities):
return sum(v * p for v, p in zip(values, probabilities))
def variance(values, probabilities):
mu = expected_value(values, probabilities)
return sum(p * (v - mu) ** 2 for v, p in zip(values, probabilities))
die_values = [1, 2, 3, 4, 5, 6]
die_probs = [1/6] * 6
mu = expected_value(die_values, die_probs)
var = variance(die_values, die_probs)
print(f"Die: E[X] = {mu:.4f}, Var(X) = {var:.4f}, SD = {var**0.5:.4f}")Step 4: Sampling from distributions
def sample_bernoulli(p, n=1):
return [1 if random.random() < p else 0 for _ in range(n)]
def sample_categorical(probs, n=1):
cumulative = []
total = 0
for p in probs:
total += p
cumulative.append(total)
samples = []
for _ in range(n):
r = random.random()
for i, c in enumerate(cumulative):
if r <= c:
samples.append(i)
break
return samples
def sample_normal_box_muller(mu, sigma, n=1):
samples = []
for _ in range(n):
u1 = random.random()
u2 = random.random()
z = math.sqrt(-2 * math.log(u1)) * math.cos(2 * math.pi * u2)
samples.append(mu + sigma * z)
return samplesStep 5: Softmax and log probabilities
def softmax(logits):
max_logit = max(logits)
shifted = [z - max_logit for z in logits]
exps = [math.exp(z) for z in shifted]
total = sum(exps)
return [e / total for e in exps]
def log_softmax(logits):
max_logit = max(logits)
shifted = [z - max_logit for z in logits]
log_sum_exp = max_logit + math.log(sum(math.exp(z) for z in shifted))
return [z - log_sum_exp for z in logits]
def cross_entropy_loss(logits, target_index):
log_probs = log_softmax(logits)
return -log_probs[target_index]Step 6: Central Limit Theorem demonstration
def demonstrate_clt(dist_fn, n_samples, n_averages):
averages = []
for _ in range(n_averages):
samples = [dist_fn() for _ in range(n_samples)]
averages.append(sum(samples) / len(samples))
return averagesStep 7: Visualization
import matplotlib.pyplot as plt
xs = [mu + sigma * (i - 500) / 100 for i in range(1001)]
ys = [normal_pdf(x, mu, sigma) for x, mu, sigma in ...]
plt.plot(xs, ys)Full implementations with all visualizations are in code/probability.py.
Use It
With NumPy and SciPy, everything above is one-liners:
import numpy as np
from scipy import stats
normal = stats.norm(loc=0, scale=1)
samples = normal.rvs(size=10000)
print(f"Mean: {np.mean(samples):.4f}, Std: {np.std(samples):.4f}")
print(f"P(X < 1.96) = {normal.cdf(1.96):.4f}")
logits = np.array([2.0, 1.0, 0.1])
from scipy.special import softmax, log_softmax
probs = softmax(logits)
log_probs = log_softmax(logits)
print(f"Softmax: {probs}")
print(f"Log-softmax: {log_probs}")You built these from scratch. Now you know what the library calls are doing.
Exercises
- Implement inverse transform sampling for the exponential distribution. Verify by sampling 10,000 values and comparing the histogram to the true PDF.
- Build a joint distribution table for two loaded dice. Compute the marginal distributions and check whether the dice are independent.
- Compute the cross-entropy loss for a 5-class classifier that outputs logits
[2.0, 0.5, -1.0, 3.0, 0.1]when the correct class is index 3. Then verify your answer with PyTorch'snn.CrossEntropyLoss. - Write a function that takes a list of log probabilities and returns the most likely sequence, the total log probability, and the equivalent raw probability. Test it with a sentence of 50 words where each word has probability 0.01.
Hints
Exercise 1. Draw u from a uniform distribution and return $-\ln(1 - u) / \lambda$. Check that the sample mean is close to $1/\lambda$ and that the histogram bins follow $\lambda e^{-\lambda x}$.
Exercise 2. Fill a 6x6 table where every cell is a probability and the sum of all cells is 1, then call joint_to_marginals. Use check_independence: the dice are independent only if every cell equals the product of its two marginals.
Exercise 3. Apply softmax to the logits and take $-\ln$ of the probability at index 3. The result must be close to 0.421, which is the value nn.CrossEntropyLoss gives for the same logits and target.
Exercise 4. Add the log probabilities; do not multiply raw probabilities. For 50 words with probability 0.01, the total must be $50 \ln(0.01) \approx -230.26$, and the raw probability is $10^{-100}$.
Key Terms
| Term | What people say | What it actually means |
|---|---|---|
| Sample space | "All the possibilities" | The set $S$ of every possible outcome of an experiment |
| PMF | "The probability function" | A function that gives the exact probability of each discrete outcome, summing to 1 |
| "The probability curve" | A density function for continuous variables. Integrate it over an interval to get probability | |
| Conditional probability | "Probability given something" | $P(A \mid B) = P(A \text{ and } B) / P(B)$. The foundation of Bayesian thinking and Bayes' theorem |
| Independence | "They don't affect each other" | $P(A \text{ and } B) = P(A) \cdot P(B)$. Knowing one event tells you nothing about the other |
| Expected value | "The average" | The probability-weighted sum of all outcomes. The loss function is an expected value |
| Variance | "How spread out" | The expected squared deviation from the mean. High variance = noisy, unstable estimates |
| Normal distribution | "The bell curve" | $f(x) = (1/\sqrt{2\pi\sigma^2}) \cdot \exp(-(x-\mu)^2/(2\sigma^2))$. Appears everywhere due to the CLT |
| Central Limit Theorem | "Averages become normal" | The mean of many independent samples converges to a normal distribution regardless of the source |
| Joint distribution | "Two variables together" | $P(X, Y)$ describes the probability of every combination of $X$ and $Y$ outcomes |
| Marginal distribution | "Sum out the other variable" | $P(X) = \sum_y P(X, Y)$. Recovers one variable's distribution from the joint |
| Log probability | "Log of the probability" | $\log P(x)$. Turns products into sums, preventing numerical underflow in long sequences |
| Softmax | "Turn scores into probabilities" | $\text{softmax}(z_i) = \exp(z_i) / \sum_j \exp(z_j)$. Maps real-valued logits to a valid probability distribution |
| Cross-entropy | "The loss function" | $-\sum p_{\text{true}} \cdot \log(p_{\text{predicted}})$. Measures how different two distributions are. Lower is better |
| Logits | "Raw model outputs" | Unnormalized scores before softmax. Named after the logistic function |
| Sampling | "Drawing random values" | Generating values according to a probability distribution. How models generate output |
Further Reading
- 3Blue1Brown: But what is the Central Limit Theorem? - visual proof of why averages become normal
- Stanford CS229 Probability Review - concise reference covering everything here and more
- The Log-Sum-Exp Trick - why numerical stability matters and how to achieve it
Check what you learned
5 questions. Answer each one to see why.
What is the difference between a probability mass function (PMF) and a probability density function (PDF)?
The full code
Every file for this lesson. Copy it, run it, then change it.
# Probability + distributions in Julia. Hand-written PMFs, PDFs,
# samplers (Bernoulli, Categorical, Uniform, Normal via Box-Muller),
# softmax + log-softmax + cross-entropy, marginals, central limit demo.
# Stdlib only. Sources:
# https://docs.julialang.org/en/v1/stdlib/Random/
# https://docs.julialang.org/en/v1/manual/missing/
# https://en.wikipedia.org/wiki/Box-Muller_transform
using Random
using Statistics
using Printf
factorial_int(n::Int)::Int = n <= 1 ? 1 : prod(2:n)
function combinations(n::Int, k::Int)::Int
return factorial_int(n) ÷ (factorial_int(k) * factorial_int(n - k))
end
function conditional_probability(p_a_and_b::Float64, p_b::Float64)
if p_b == 0.0
throw(ArgumentError("conditional_probability: P(B) is zero; cannot divide"))
end
return p_a_and_b / p_b
end
bernoulli_pmf(k::Int, p::Float64) = k == 1 ? p : (1 - p)
categorical_pmf(k::Int, probs::Vector{Float64}) = probs[k + 1]
function poisson_pmf(k::Int, lam::Float64)
return (lam ^ k) * exp(-lam) / factorial_int(k)
end
function uniform_pdf(x::Float64, a::Float64, b::Float64)
return a <= x <= b ? 1.0 / (b - a) : 0.0
end
function normal_pdf(x::Float64, mu::Float64, sigma::Float64)
coeff = 1.0 / (sigma * sqrt(2pi))
exponent = -0.5 * ((x - mu) / sigma) ^ 2
return coeff * exp(exponent)
end
function expected_value(values::Vector{Float64}, probs::Vector{Float64})::Float64
return sum(values .* probs)
end
function variance_of(values::Vector{Float64}, probs::Vector{Float64})::Float64
mu = expected_value(values, probs)
return sum(probs .* (values .- mu) .^ 2)
end
function sample_bernoulli(rng::AbstractRNG, p::Float64, n::Int)
return [rand(rng) < p ? 1 : 0 for _ in 1:n]
end
function sample_categorical(rng::AbstractRNG, probs::Vector{Float64}, n::Int)
cumulative = cumsum(probs)
samples = Int[]
for _ in 1:n
r = rand(rng)
idx = findfirst(c -> r <= c, cumulative)
push!(samples, idx === nothing ? length(probs) - 1 : idx - 1)
end
return samples
end
function sample_uniform(rng::AbstractRNG, a::Float64, b::Float64, n::Int)
return [a + (b - a) * rand(rng) for _ in 1:n]
end
function sample_normal_box_muller(rng::AbstractRNG, mu::Float64, sigma::Float64, n::Int)
samples = Float64[]
for _ in 1:n
# rand(rng) is in [0, 1); guard against u1 == 0 so log(u1) stays finite.
u1 = rand(rng)
while u1 == 0.0
u1 = rand(rng)
end
u2 = rand(rng)
z = sqrt(-2 * log(u1)) * cos(2pi * u2)
push!(samples, mu + sigma * z)
end
return samples
end
function softmax(logits::Vector{Float64})
m = maximum(logits)
exps = exp.(logits .- m)
return exps ./ sum(exps)
end
function log_softmax(logits::Vector{Float64})
m = maximum(logits)
shifted = logits .- m
log_sum_exp = m + log(sum(exp.(shifted)))
return logits .- log_sum_exp
end
function cross_entropy_loss(logits::Vector{Float64}, target_index::Int)
return -log_softmax(logits)[target_index + 1]
end
function joint_to_marginals(joint::Matrix{Float64})
marginal_x = vec(sum(joint, dims=2))
marginal_y = vec(sum(joint, dims=1))
return marginal_x, marginal_y
end
function check_independence(joint::Matrix{Float64},
marginal_x::Vector{Float64},
marginal_y::Vector{Float64};
tol::Float64=1e-9)::Bool
for i in eachindex(marginal_x), j in eachindex(marginal_y)
if abs(joint[i, j] - marginal_x[i] * marginal_y[j]) > tol
return false
end
end
return true
end
function demonstrate_clt(rng::AbstractRNG, n_per_sample::Int, n_averages::Int)
averages = Float64[]
for _ in 1:n_averages
samples = rand(rng, n_per_sample)
push!(averages, mean(samples))
end
return averages
end
function main()
rng = MersenneTwister(42)
println("=" ^ 60)
println("PROBABILITY AND DISTRIBUTIONS")
println("=" ^ 60)
println("\n--- Conditional Probability ---")
p_king_given_face = conditional_probability(4 / 52, 12 / 52)
@printf("P(King | Face card) = %.4f\n", p_king_given_face)
println("\n--- PMF: Bernoulli (p=0.7) ---")
for k in 0:1
@printf(" P(X=%d) = %.4f\n", k, bernoulli_pmf(k, 0.7))
end
println("\n--- PMF: Categorical ---")
cat_probs = Float64[0.1, 0.3, 0.4, 0.2]
for k in 0:(length(cat_probs) - 1)
@printf(" P(X=%d) = %.4f\n", k, categorical_pmf(k, cat_probs))
end
println("\n--- PMF: Poisson (lambda=3) ---")
for k in 0:9
@printf(" P(X=%d) = %.4f\n", k, poisson_pmf(k, 3.0))
end
println("\n--- PDF: Normal (mu=0, sigma=1) ---")
for x in -3.0:1.0:3.0
@printf(" f(%+.0f) = %.4f\n", x, normal_pdf(x, 0.0, 1.0))
end
println("\n--- Expected Value & Variance ---")
die_values = Float64[1, 2, 3, 4, 5, 6]
die_probs = fill(1 / 6, 6)
mu = expected_value(die_values, die_probs)
var = variance_of(die_values, die_probs)
@printf(" Fair die: E[X] = %.4f, Var(X) = %.4f, SD = %.4f\n", mu, var, sqrt(var))
println("\n--- Sampling: Bernoulli (p=0.3, n=20) ---")
bern = sample_bernoulli(rng, 0.3, 20)
println(" Samples: $bern")
@printf(" Empirical mean: %.4f (expected 0.3)\n", mean(bern))
println("\n--- Sampling: Categorical ---")
cat_samples = sample_categorical(rng, Float64[0.1, 0.3, 0.4, 0.2], 1000)
counts = [count(==(i), cat_samples) for i in 0:3]
println(" Counts from 1000 samples: $counts")
println(" Empirical: $(round.(counts ./ 1000, digits=4))")
println(" Expected: [0.1, 0.3, 0.4, 0.2]")
println("\n--- Sampling: Normal (Box-Muller) ---")
norm = sample_normal_box_muller(rng, 0.0, 1.0, 10000)
sample_mean = mean(norm)
sample_var = var_of_samples(norm)
println(" 10000 samples from N(0, 1):")
@printf(" Sample mean: %.4f (expected 0)\n", sample_mean)
@printf(" Sample var: %.4f (expected 1)\n", sample_var)
println("\n--- Softmax ---")
logits = Float64[2.0, 1.0, 0.1]
probs = softmax(logits)
println(" Logits: $logits")
println(" Softmax: $(round.(probs, digits=4))")
@printf(" Sum: %.4f\n", sum(probs))
println("\n--- Softmax with large logits (stability test) ---")
large_logits = Float64[100, 101, 102]
probs_large = softmax(large_logits)
println(" Logits: $large_logits")
println(" Softmax: $(round.(probs_large, digits=4))")
println(" (No overflow because we subtract max before exp)")
println("\n--- Log Probabilities ---")
lp = log_softmax(logits)
println(" Logits: $logits")
println(" Log-softmax: $(round.(lp, digits=4))")
println(" Verify exp: $(round.(exp.(lp), digits=4))")
println("\n--- Cross-Entropy Loss ---")
ce = cross_entropy_loss(Float64[2.0, 1.0, 0.1], 0)
println(" Logits: [2.0, 1.0, 0.1], target: 0")
@printf(" Cross-entropy loss: %.4f\n", ce)
println("\n--- Why log probabilities matter ---")
word_prob = 0.01
n_words = 50
raw_product = word_prob ^ n_words
log_sum = n_words * log(word_prob)
@printf(" P(word)^%d = %.2e\n", n_words, raw_product)
@printf(" Log sum: %.4f (stable)\n", log_sum)
@printf(" Recovered: %.2e\n", exp(log_sum))
println("\n--- Joint & Marginal Distributions ---")
joint = Float64[0.40 0.10; 0.05 0.45]
mx, my = joint_to_marginals(joint)
println(" Joint (weather x umbrella):")
@printf(" Sun, no umbrella: %.2f\n", joint[1, 1])
@printf(" Sun, umbrella: %.2f\n", joint[1, 2])
@printf(" Rain, no umbrella: %.2f\n", joint[2, 1])
@printf(" Rain, umbrella: %.2f\n", joint[2, 2])
println(" Marginal X (weather): $mx")
println(" Marginal Y (umbrella): $my")
println(" Independent? $(check_independence(joint, mx, my))")
println("\n--- Central Limit Theorem ---")
println(" Averaging uniform [0, 1) samples:")
for n in [1, 2, 5, 30]
avgs = demonstrate_clt(rng, n, 10000)
@printf(" n=%2d: mean=%.4f, std=%.4f\n", n, mean(avgs), std_of_samples(avgs))
end
println(" As n grows, std shrinks and distribution approaches normal.")
println("\n" * "=" ^ 60)
println("All probability computations complete.")
println("=" ^ 60)
end
function var_of_samples(xs::Vector{Float64})::Float64
m = mean(xs)
return sum((xs .- m) .^ 2) / length(xs)
end
function std_of_samples(xs::Vector{Float64})::Float64
return sqrt(var_of_samples(xs))
end
if abspath(PROGRAM_FILE) == @__FILE__
main()
end