Get Started

Probability and Distributions

Probability is the language AI uses to express uncertainty.

75 minutesLearnPythonFree lesson

Learning Objectives

  • Implement PMFs and PDFs from scratch for Bernoulli, categorical, Poisson, uniform, and normal distributions
  • Compute expected value, variance, and use the Central Limit Theorem to explain why Gaussians dominate
  • Build softmax and log-softmax functions with the numerical stability trick (subtract max logit)
  • Calculate cross-entropy loss from logits and connect it to negative log-likelihood

The Problem

A classifier outputs [0.03, 0.91, 0.06]. A language model picks the next word from 50,000 candidates. A diffusion model generates images by sampling from learned distributions. All of these are probability in action.

Every prediction a model makes is a probability distribution. Every loss function measures how far the predicted distribution is from the true one. Every training step adjusts parameters to make one distribution look more like another. Without probability, you cannot read a single ML paper, debug a single model, or understand why your training loss is NaN.

The Concept

Events, Sample Spaces, and Probability

The sample space $S$ is the set of all possible outcomes. An event is a subset of the sample space. Probability maps events to numbers between 0 and 1.

Code
Coin flip:
  S = {H, T}
  P(H) = 0.5,  P(T) = 0.5

Single die roll:
  S = {1, 2, 3, 4, 5, 6}
  P(even) = P({2, 4, 6}) = 3/6 = 0.5

Three axioms define all of probability:

  1. $P(A) \ge 0$ for any event $A$
  2. $P(S) = 1$ (something always happens)
  3. $P(A \text{ or } B) = P(A) + P(B)$ when $A$ and $B$ cannot both occur

Everything else (Bayes' theorem, expectations, distributions) follows from these three rules.

Conditional Probability and Independence

$P(A \mid B)$ is the probability of $A$ given that $B$ happened.

Code
P(A|B) = P(A and B) / P(B)

Example: deck of cards
  P(King | Face card) = P(King and Face card) / P(Face card)
                      = (4/52) / (12/52)
                      = 4/12 = 1/3

Two events are independent when knowing one tells you nothing about the other:

Coin flips are independent. Drawing cards without replacement is not.

Probability Mass Functions vs Probability Density Functions

Discrete random variables have a probability mass function (PMF). Each outcome has a specific probability that you can read off directly.

Code
PMF: P(X = k)

Fair die:
  P(X = 1) = 1/6
  P(X = 2) = 1/6
  ...
  P(X = 6) = 1/6

  Sum of all probabilities = 1

Continuous random variables have a probability density function (PDF). The density at a single point is not a probability. Probability comes from integrating the density over an interval.

Code
PDF: f(x)

P(a <= X <= b) = integral of f(x) from a to b

f(x) can be greater than 1 (density, not probability)
integral from -inf to +inf of f(x) dx = 1

This distinction matters in ML. Classification outputs are PMFs (discrete choices). VAE latent spaces use PDFs (continuous).

Common Distributions

Bernoulli: one trial, two outcomes. Models binary classification.

Categorical: one trial, $k$ outcomes. Models multi-class classification (softmax output).

Uniform: all outcomes equally likely. Used for random initialization.

Normal (Gaussian): the bell curve. Parameterized by mean ($\mu$) and variance ($\sigma^2$).

Code
f(x) = (1 / sqrt(2*pi*sigma^2)) * exp(-(x - mu)^2 / (2*sigma^2))

Standard normal: mu = 0, sigma = 1
  68% of data within 1 sigma
  95% within 2 sigma
  99.7% within 3 sigma

Poisson: counts of rare events in a fixed interval. Models event rates.

Expected Value and Variance

Expected value is the weighted average outcome.

Variance measures spread around the mean.

In ML, expected value appears as the loss function (average loss over the data distribution). Variance tells you about model stability. High variance in gradients means noisy training.

Joint and Marginal Distributions

A joint distribution $P(X, Y)$ describes two random variables together.

Joint PMF example ($X$ = weather, $Y$ = umbrella):

$Y=0$ (no umbrella)$Y=1$ (umbrella)Marginal $P(X)$
$X=0$ (sun)0.400.10$P(X=0) = 0.50$
$X=1$ (rain)0.050.45$P(X=1) = 0.50$
Marginal $P(Y)$$P(Y=0) = 0.45$$P(Y=1) = 0.55$1.00

The marginal distribution sums out the other variable:

The row and column totals in the table above are the marginals.

Why the Normal Distribution Shows Up Everywhere

The Central Limit Theorem: the sum (or average) of many independent random variables converges to a normal distribution, regardless of the original distribution.

Code
Roll 1 die:  uniform distribution (flat)
Average of 2 dice:  triangular (peaked)
Average of 30 dice: nearly perfect bell curve

This works for ANY starting distribution.

This is why:

  • Measurement errors are approximately normal (many small independent sources)
  • Weight initializations in neural networks use normal distributions
  • Gradient noise in SGD is approximately normal (sum of many sample gradients)
  • The normal distribution is the maximum entropy distribution for a given mean and variance

Log Probabilities

Raw probabilities cause numerical problems. Multiplying many small probabilities together quickly underflows to zero.

Log probabilities fix this. Multiplications become additions.

Rules:

  • $\log(a \cdot b) = \log(a) + \log(b)$
  • log probabilities are always $\le 0$ (since $0 < P \le 1$)
  • More negative = less likely
  • Cross-entropy loss is the negative log probability of the correct class

Softmax as a Probability Distribution

Neural networks output raw scores (logits). Softmax converts them into a valid probability distribution.

Code
softmax(z_i) = exp(z_i) / sum(exp(z_j) for all j)

Properties:
  - All outputs are in (0, 1)
  - All outputs sum to 1
  - Preserves relative ordering of inputs
  - exp() amplifies differences between logits

The softmax trick: subtract the max logit before exponentiating to prevent overflow.

Code
z = [100, 101, 102]
exp(102) = overflow

z_shifted = z - max(z) = [-2, -1, 0]
exp(0) = 1  (safe)

Same result, no overflow.

Log-softmax combines softmax and log for numerical stability. PyTorch uses this internally for cross-entropy loss.

Sampling

Sampling means drawing random values from a distribution. In ML:

  • Dropout randomly samples which neurons to zero out
  • Data augmentation samples random transformations
  • Language models sample the next token from the predicted distribution
  • Diffusion models sample noise and progressively denoise

Sampling from arbitrary distributions requires techniques like inverse transform sampling, rejection sampling, or the reparameterization trick (used in VAEs).

Build It

Step 1: Probability basics

Python
import math
import random

def factorial(n):
    result = 1
    for i in range(2, n + 1):
        result *= i
    return result

def combinations(n, k):
    return factorial(n) // (factorial(k) * factorial(n - k))

def conditional_probability(p_a_and_b, p_b):
    return p_a_and_b / p_b

p_king_given_face = conditional_probability(4/52, 12/52)
print(f"P(King | Face card) = {p_king_given_face:.4f}")

Step 2: PMF and PDF from scratch

Python
def bernoulli_pmf(k, p):
    return p if k == 1 else (1 - p)

def categorical_pmf(k, probs):
    return probs[k]

def poisson_pmf(k, lam):
    return (lam ** k) * math.exp(-lam) / factorial(k)

def uniform_pdf(x, a, b):
    if a <= x <= b:
        return 1.0 / (b - a)
    return 0.0

def normal_pdf(x, mu, sigma):
    coeff = 1.0 / (sigma * math.sqrt(2 * math.pi))
    exponent = -0.5 * ((x - mu) / sigma) ** 2
    return coeff * math.exp(exponent)

Step 3: Expected value and variance

Python
def expected_value(values, probabilities):
    return sum(v * p for v, p in zip(values, probabilities))

def variance(values, probabilities):
    mu = expected_value(values, probabilities)
    return sum(p * (v - mu) ** 2 for v, p in zip(values, probabilities))

die_values = [1, 2, 3, 4, 5, 6]
die_probs = [1/6] * 6
mu = expected_value(die_values, die_probs)
var = variance(die_values, die_probs)
print(f"Die: E[X] = {mu:.4f}, Var(X) = {var:.4f}, SD = {var**0.5:.4f}")

Step 4: Sampling from distributions

Python
def sample_bernoulli(p, n=1):
    return [1 if random.random() < p else 0 for _ in range(n)]

def sample_categorical(probs, n=1):
    cumulative = []
    total = 0
    for p in probs:
        total += p
        cumulative.append(total)
    samples = []
    for _ in range(n):
        r = random.random()
        for i, c in enumerate(cumulative):
            if r <= c:
                samples.append(i)
                break
    return samples

def sample_normal_box_muller(mu, sigma, n=1):
    samples = []
    for _ in range(n):
        u1 = random.random()
        u2 = random.random()
        z = math.sqrt(-2 * math.log(u1)) * math.cos(2 * math.pi * u2)
        samples.append(mu + sigma * z)
    return samples

Step 5: Softmax and log probabilities

Python
def softmax(logits):
    max_logit = max(logits)
    shifted = [z - max_logit for z in logits]
    exps = [math.exp(z) for z in shifted]
    total = sum(exps)
    return [e / total for e in exps]

def log_softmax(logits):
    max_logit = max(logits)
    shifted = [z - max_logit for z in logits]
    log_sum_exp = max_logit + math.log(sum(math.exp(z) for z in shifted))
    return [z - log_sum_exp for z in logits]

def cross_entropy_loss(logits, target_index):
    log_probs = log_softmax(logits)
    return -log_probs[target_index]

Step 6: Central Limit Theorem demonstration

Python
def demonstrate_clt(dist_fn, n_samples, n_averages):
    averages = []
    for _ in range(n_averages):
        samples = [dist_fn() for _ in range(n_samples)]
        averages.append(sum(samples) / len(samples))
    return averages

Step 7: Visualization

Python
import matplotlib.pyplot as plt

xs = [mu + sigma * (i - 500) / 100 for i in range(1001)]
ys = [normal_pdf(x, mu, sigma) for x, mu, sigma in ...]
plt.plot(xs, ys)

Full implementations with all visualizations are in code/probability.py.

Use It

With NumPy and SciPy, everything above is one-liners:

Python
import numpy as np
from scipy import stats

normal = stats.norm(loc=0, scale=1)
samples = normal.rvs(size=10000)
print(f"Mean: {np.mean(samples):.4f}, Std: {np.std(samples):.4f}")
print(f"P(X < 1.96) = {normal.cdf(1.96):.4f}")

logits = np.array([2.0, 1.0, 0.1])
from scipy.special import softmax, log_softmax
probs = softmax(logits)
log_probs = log_softmax(logits)
print(f"Softmax: {probs}")
print(f"Log-softmax: {log_probs}")

You built these from scratch. Now you know what the library calls are doing.

Exercises

  1. Implement inverse transform sampling for the exponential distribution. Verify by sampling 10,000 values and comparing the histogram to the true PDF.
  2. Build a joint distribution table for two loaded dice. Compute the marginal distributions and check whether the dice are independent.
  3. Compute the cross-entropy loss for a 5-class classifier that outputs logits [2.0, 0.5, -1.0, 3.0, 0.1] when the correct class is index 3. Then verify your answer with PyTorch's nn.CrossEntropyLoss.
  4. Write a function that takes a list of log probabilities and returns the most likely sequence, the total log probability, and the equivalent raw probability. Test it with a sentence of 50 words where each word has probability 0.01.

Hints

Exercise 1. Draw u from a uniform distribution and return $-\ln(1 - u) / \lambda$. Check that the sample mean is close to $1/\lambda$ and that the histogram bins follow $\lambda e^{-\lambda x}$.

Exercise 2. Fill a 6x6 table where every cell is a probability and the sum of all cells is 1, then call joint_to_marginals. Use check_independence: the dice are independent only if every cell equals the product of its two marginals.

Exercise 3. Apply softmax to the logits and take $-\ln$ of the probability at index 3. The result must be close to 0.421, which is the value nn.CrossEntropyLoss gives for the same logits and target.

Exercise 4. Add the log probabilities; do not multiply raw probabilities. For 50 words with probability 0.01, the total must be $50 \ln(0.01) \approx -230.26$, and the raw probability is $10^{-100}$.

Key Terms

TermWhat people sayWhat it actually means
Sample space"All the possibilities"The set $S$ of every possible outcome of an experiment
PMF"The probability function"A function that gives the exact probability of each discrete outcome, summing to 1
PDF"The probability curve"A density function for continuous variables. Integrate it over an interval to get probability
Conditional probability"Probability given something"$P(A \mid B) = P(A \text{ and } B) / P(B)$. The foundation of Bayesian thinking and Bayes' theorem
Independence"They don't affect each other"$P(A \text{ and } B) = P(A) \cdot P(B)$. Knowing one event tells you nothing about the other
Expected value"The average"The probability-weighted sum of all outcomes. The loss function is an expected value
Variance"How spread out"The expected squared deviation from the mean. High variance = noisy, unstable estimates
Normal distribution"The bell curve"$f(x) = (1/\sqrt{2\pi\sigma^2}) \cdot \exp(-(x-\mu)^2/(2\sigma^2))$. Appears everywhere due to the CLT
Central Limit Theorem"Averages become normal"The mean of many independent samples converges to a normal distribution regardless of the source
Joint distribution"Two variables together"$P(X, Y)$ describes the probability of every combination of $X$ and $Y$ outcomes
Marginal distribution"Sum out the other variable"$P(X) = \sum_y P(X, Y)$. Recovers one variable's distribution from the joint
Log probability"Log of the probability"$\log P(x)$. Turns products into sums, preventing numerical underflow in long sequences
Softmax"Turn scores into probabilities"$\text{softmax}(z_i) = \exp(z_i) / \sum_j \exp(z_j)$. Maps real-valued logits to a valid probability distribution
Cross-entropy"The loss function"$-\sum p_{\text{true}} \cdot \log(p_{\text{predicted}})$. Measures how different two distributions are. Lower is better
Logits"Raw model outputs"Unnormalized scores before softmax. Named after the logistic function
Sampling"Drawing random values"Generating values according to a probability distribution. How models generate output

Further Reading

Check what you learned

5 questions. Answer each one to see why.

Question 1 of 50 correct

What is the difference between a probability mass function (PMF) and a probability density function (PDF)?

The full code

Every file for this lesson. Copy it, run it, then change it.

main.jl
# Probability + distributions in Julia. Hand-written PMFs, PDFs,
# samplers (Bernoulli, Categorical, Uniform, Normal via Box-Muller),
# softmax + log-softmax + cross-entropy, marginals, central limit demo.
# Stdlib only. Sources:
#   https://docs.julialang.org/en/v1/stdlib/Random/
#   https://docs.julialang.org/en/v1/manual/missing/
#   https://en.wikipedia.org/wiki/Box-Muller_transform

using Random
using Statistics
using Printf


factorial_int(n::Int)::Int = n <= 1 ? 1 : prod(2:n)


function combinations(n::Int, k::Int)::Int
    return factorial_int(n) ÷ (factorial_int(k) * factorial_int(n - k))
end


function conditional_probability(p_a_and_b::Float64, p_b::Float64)
    if p_b == 0.0
        throw(ArgumentError("conditional_probability: P(B) is zero; cannot divide"))
    end
    return p_a_and_b / p_b
end


bernoulli_pmf(k::Int, p::Float64) = k == 1 ? p : (1 - p)


categorical_pmf(k::Int, probs::Vector{Float64}) = probs[k + 1]


function poisson_pmf(k::Int, lam::Float64)
    return (lam ^ k) * exp(-lam) / factorial_int(k)
end


function uniform_pdf(x::Float64, a::Float64, b::Float64)
    return a <= x <= b ? 1.0 / (b - a) : 0.0
end


function normal_pdf(x::Float64, mu::Float64, sigma::Float64)
    coeff = 1.0 / (sigma * sqrt(2pi))
    exponent = -0.5 * ((x - mu) / sigma) ^ 2
    return coeff * exp(exponent)
end


function expected_value(values::Vector{Float64}, probs::Vector{Float64})::Float64
    return sum(values .* probs)
end


function variance_of(values::Vector{Float64}, probs::Vector{Float64})::Float64
    mu = expected_value(values, probs)
    return sum(probs .* (values .- mu) .^ 2)
end


function sample_bernoulli(rng::AbstractRNG, p::Float64, n::Int)
    return [rand(rng) < p ? 1 : 0 for _ in 1:n]
end


function sample_categorical(rng::AbstractRNG, probs::Vector{Float64}, n::Int)
    cumulative = cumsum(probs)
    samples = Int[]
    for _ in 1:n
        r = rand(rng)
        idx = findfirst(c -> r <= c, cumulative)
        push!(samples, idx === nothing ? length(probs) - 1 : idx - 1)
    end
    return samples
end


function sample_uniform(rng::AbstractRNG, a::Float64, b::Float64, n::Int)
    return [a + (b - a) * rand(rng) for _ in 1:n]
end


function sample_normal_box_muller(rng::AbstractRNG, mu::Float64, sigma::Float64, n::Int)
    samples = Float64[]
    for _ in 1:n
        # rand(rng) is in [0, 1); guard against u1 == 0 so log(u1) stays finite.
        u1 = rand(rng)
        while u1 == 0.0
            u1 = rand(rng)
        end
        u2 = rand(rng)
        z = sqrt(-2 * log(u1)) * cos(2pi * u2)
        push!(samples, mu + sigma * z)
    end
    return samples
end


function softmax(logits::Vector{Float64})
    m = maximum(logits)
    exps = exp.(logits .- m)
    return exps ./ sum(exps)
end


function log_softmax(logits::Vector{Float64})
    m = maximum(logits)
    shifted = logits .- m
    log_sum_exp = m + log(sum(exp.(shifted)))
    return logits .- log_sum_exp
end


function cross_entropy_loss(logits::Vector{Float64}, target_index::Int)
    return -log_softmax(logits)[target_index + 1]
end


function joint_to_marginals(joint::Matrix{Float64})
    marginal_x = vec(sum(joint, dims=2))
    marginal_y = vec(sum(joint, dims=1))
    return marginal_x, marginal_y
end


function check_independence(joint::Matrix{Float64},
                            marginal_x::Vector{Float64},
                            marginal_y::Vector{Float64};
                            tol::Float64=1e-9)::Bool
    for i in eachindex(marginal_x), j in eachindex(marginal_y)
        if abs(joint[i, j] - marginal_x[i] * marginal_y[j]) > tol
            return false
        end
    end
    return true
end


function demonstrate_clt(rng::AbstractRNG, n_per_sample::Int, n_averages::Int)
    averages = Float64[]
    for _ in 1:n_averages
        samples = rand(rng, n_per_sample)
        push!(averages, mean(samples))
    end
    return averages
end


function main()
    rng = MersenneTwister(42)

    println("=" ^ 60)
    println("PROBABILITY AND DISTRIBUTIONS")
    println("=" ^ 60)

    println("\n--- Conditional Probability ---")
    p_king_given_face = conditional_probability(4 / 52, 12 / 52)
    @printf("P(King | Face card) = %.4f\n", p_king_given_face)

    println("\n--- PMF: Bernoulli (p=0.7) ---")
    for k in 0:1
        @printf("  P(X=%d) = %.4f\n", k, bernoulli_pmf(k, 0.7))
    end

    println("\n--- PMF: Categorical ---")
    cat_probs = Float64[0.1, 0.3, 0.4, 0.2]
    for k in 0:(length(cat_probs) - 1)
        @printf("  P(X=%d) = %.4f\n", k, categorical_pmf(k, cat_probs))
    end

    println("\n--- PMF: Poisson (lambda=3) ---")
    for k in 0:9
        @printf("  P(X=%d) = %.4f\n", k, poisson_pmf(k, 3.0))
    end

    println("\n--- PDF: Normal (mu=0, sigma=1) ---")
    for x in -3.0:1.0:3.0
        @printf("  f(%+.0f) = %.4f\n", x, normal_pdf(x, 0.0, 1.0))
    end

    println("\n--- Expected Value & Variance ---")
    die_values = Float64[1, 2, 3, 4, 5, 6]
    die_probs = fill(1 / 6, 6)
    mu = expected_value(die_values, die_probs)
    var = variance_of(die_values, die_probs)
    @printf("  Fair die: E[X] = %.4f, Var(X) = %.4f, SD = %.4f\n", mu, var, sqrt(var))

    println("\n--- Sampling: Bernoulli (p=0.3, n=20) ---")
    bern = sample_bernoulli(rng, 0.3, 20)
    println("  Samples: $bern")
    @printf("  Empirical mean: %.4f (expected 0.3)\n", mean(bern))

    println("\n--- Sampling: Categorical ---")
    cat_samples = sample_categorical(rng, Float64[0.1, 0.3, 0.4, 0.2], 1000)
    counts = [count(==(i), cat_samples) for i in 0:3]
    println("  Counts from 1000 samples: $counts")
    println("  Empirical: $(round.(counts ./ 1000, digits=4))")
    println("  Expected:  [0.1, 0.3, 0.4, 0.2]")

    println("\n--- Sampling: Normal (Box-Muller) ---")
    norm = sample_normal_box_muller(rng, 0.0, 1.0, 10000)
    sample_mean = mean(norm)
    sample_var = var_of_samples(norm)
    println("  10000 samples from N(0, 1):")
    @printf("  Sample mean: %.4f (expected 0)\n", sample_mean)
    @printf("  Sample var:  %.4f (expected 1)\n", sample_var)

    println("\n--- Softmax ---")
    logits = Float64[2.0, 1.0, 0.1]
    probs = softmax(logits)
    println("  Logits:  $logits")
    println("  Softmax: $(round.(probs, digits=4))")
    @printf("  Sum:     %.4f\n", sum(probs))

    println("\n--- Softmax with large logits (stability test) ---")
    large_logits = Float64[100, 101, 102]
    probs_large = softmax(large_logits)
    println("  Logits:  $large_logits")
    println("  Softmax: $(round.(probs_large, digits=4))")
    println("  (No overflow because we subtract max before exp)")

    println("\n--- Log Probabilities ---")
    lp = log_softmax(logits)
    println("  Logits:      $logits")
    println("  Log-softmax: $(round.(lp, digits=4))")
    println("  Verify exp:  $(round.(exp.(lp), digits=4))")

    println("\n--- Cross-Entropy Loss ---")
    ce = cross_entropy_loss(Float64[2.0, 1.0, 0.1], 0)
    println("  Logits: [2.0, 1.0, 0.1], target: 0")
    @printf("  Cross-entropy loss: %.4f\n", ce)

    println("\n--- Why log probabilities matter ---")
    word_prob = 0.01
    n_words = 50
    raw_product = word_prob ^ n_words
    log_sum = n_words * log(word_prob)
    @printf("  P(word)^%d = %.2e\n", n_words, raw_product)
    @printf("  Log sum: %.4f (stable)\n", log_sum)
    @printf("  Recovered: %.2e\n", exp(log_sum))

    println("\n--- Joint & Marginal Distributions ---")
    joint = Float64[0.40 0.10; 0.05 0.45]
    mx, my = joint_to_marginals(joint)
    println("  Joint (weather x umbrella):")
    @printf("    Sun, no umbrella: %.2f\n", joint[1, 1])
    @printf("    Sun, umbrella:    %.2f\n", joint[1, 2])
    @printf("    Rain, no umbrella: %.2f\n", joint[2, 1])
    @printf("    Rain, umbrella:    %.2f\n", joint[2, 2])
    println("  Marginal X (weather):  $mx")
    println("  Marginal Y (umbrella): $my")
    println("  Independent? $(check_independence(joint, mx, my))")

    println("\n--- Central Limit Theorem ---")
    println("  Averaging uniform [0, 1) samples:")
    for n in [1, 2, 5, 30]
        avgs = demonstrate_clt(rng, n, 10000)
        @printf("    n=%2d: mean=%.4f, std=%.4f\n", n, mean(avgs), std_of_samples(avgs))
    end
    println("  As n grows, std shrinks and distribution approaches normal.")

    println("\n" * "=" ^ 60)
    println("All probability computations complete.")
    println("=" ^ 60)
end


function var_of_samples(xs::Vector{Float64})::Float64
    m = mean(xs)
    return sum((xs .- m) .^ 2) / length(xs)
end


function std_of_samples(xs::Vector{Float64})::Float64
    return sqrt(var_of_samples(xs))
end


if abspath(PROGRAM_FILE) == @__FILE__
    main()
end